From webhook-mailer at python.org Fri Oct 1 03:36:33 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 07:36:33 -0000 Subject: [Python-checkins] bpo-45229: Make datetime tests discoverable (GH-28615) Message-ID: https://github.com/python/cpython/commit/ef7c7294e8404d844c1add892a8f6684e6cf4f31 commit: ef7c7294e8404d844c1add892a8f6684e6cf4f31 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T00:36:17-07:00 summary: bpo-45229: Make datetime tests discoverable (GH-28615) (cherry picked from commit d441437ee71ae174c008c23308b749b91020ba77) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_datetime.py diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py index bdb9f02e5756a2..7f9094fa7bd4e6 100644 --- a/Lib/test/test_datetime.py +++ b/Lib/test/test_datetime.py @@ -1,59 +1,57 @@ import unittest import sys -from test.support import run_unittest from test.support.import_helper import import_fresh_module TESTS = 'test.datetimetester' -try: - pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], - blocked=['_datetime']) - fast_tests = import_fresh_module(TESTS, fresh=['datetime', - '_datetime', '_strptime']) -finally: - # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, - # XXX: but it does not, so we have to cleanup ourselves. - for modname in ['datetime', '_datetime', '_strptime']: - sys.modules.pop(modname, None) -test_modules = [pure_tests, fast_tests] -test_suffixes = ["_Pure", "_Fast"] -# XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might -# not believe this, but in spite of all the sys.modules trickery running a _Pure -# test last will leave a mix of pure and native datetime stuff lying around. -all_test_classes = [] +def load_tests(loader, tests, pattern): + try: + pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], + blocked=['_datetime']) + fast_tests = import_fresh_module(TESTS, fresh=['datetime', + '_datetime', '_strptime']) + finally: + # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, + # XXX: but it does not, so we have to cleanup ourselves. + for modname in ['datetime', '_datetime', '_strptime']: + sys.modules.pop(modname, None) -for module, suffix in zip(test_modules, test_suffixes): - test_classes = [] - for name, cls in module.__dict__.items(): - if not isinstance(cls, type): - continue - if issubclass(cls, unittest.TestCase): - test_classes.append(cls) - elif issubclass(cls, unittest.TestSuite): - suit = cls() - test_classes.extend(type(test) for test in suit) - test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) - for cls in test_classes: - cls.__name__ += suffix - cls.__qualname__ += suffix - @classmethod - def setUpClass(cls_, module=module): - cls_._save_sys_modules = sys.modules.copy() - sys.modules[TESTS] = module - sys.modules['datetime'] = module.datetime_module - sys.modules['_strptime'] = module._strptime - @classmethod - def tearDownClass(cls_): - sys.modules.clear() - sys.modules.update(cls_._save_sys_modules) - cls.setUpClass = setUpClass - cls.tearDownClass = tearDownClass - all_test_classes.extend(test_classes) + test_modules = [pure_tests, fast_tests] + test_suffixes = ["_Pure", "_Fast"] + # XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might + # not believe this, but in spite of all the sys.modules trickery running a _Pure + # test last will leave a mix of pure and native datetime stuff lying around. + for module, suffix in zip(test_modules, test_suffixes): + test_classes = [] + for name, cls in module.__dict__.items(): + if not isinstance(cls, type): + continue + if issubclass(cls, unittest.TestCase): + test_classes.append(cls) + elif issubclass(cls, unittest.TestSuite): + suit = cls() + test_classes.extend(type(test) for test in suit) + test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) + for cls in test_classes: + cls.__name__ += suffix + cls.__qualname__ += suffix + @classmethod + def setUpClass(cls_, module=module): + cls_._save_sys_modules = sys.modules.copy() + sys.modules[TESTS] = module + sys.modules['datetime'] = module.datetime_module + sys.modules['_strptime'] = module._strptime + @classmethod + def tearDownClass(cls_): + sys.modules.clear() + sys.modules.update(cls_._save_sys_modules) + cls.setUpClass = setUpClass + cls.tearDownClass = tearDownClass + tests.addTests(loader.loadTestsFromTestCase(cls)) + return tests -def test_main(): - run_unittest(*all_test_classes) if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 1 03:55:37 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 07:55:37 -0000 Subject: [Python-checkins] bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28662) Message-ID: https://github.com/python/cpython/commit/1ee0f94d16f150356a4b9b0a39d44ba1d2d5b9fc commit: 1ee0f94d16f150356a4b9b0a39d44ba1d2d5b9fc branch: main author: Victor Stinner committer: vstinner date: 2021-10-01T09:55:28+02:00 summary: bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28662) On Unix, if the sem_clockwait() function is available in the C library (glibc 2.30 and newer), the threading.Lock.acquire() method now uses the monotonic clock (time.CLOCK_MONOTONIC) for the timeout, rather than using the system clock (time.CLOCK_REALTIME), to not be affected by system clock changes. configure now checks if the sem_clockwait() function is available. files: A Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst M Doc/whatsnew/3.11.rst M Python/thread_pthread.h M configure M configure.ac M pyconfig.h.in diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d01d2e263199a..ff376d231bafa 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -239,6 +239,16 @@ sqlite3 (Contributed by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland in :issue:`16379`.) +threading +--------- + +* On Unix, if the ``sem_clockwait()`` function is available in the C library + (glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses + the monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather + than using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected + by system clock changes. + (Contributed by Livius and Victor Stinner in :issue:`41710`.) + time ---- diff --git a/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst new file mode 100644 index 0000000000000..d8a4f9507c189 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst @@ -0,0 +1,5 @@ +On Unix, if the ``sem_clockwait()`` function is available in the C library +(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses the +monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather than +using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected by +system clock changes. Patch by Victor Stinner. diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index 3815ffae20c01..9b5e273f1a8ba 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -92,7 +92,7 @@ * mutexes and condition variables: */ #if (defined(_POSIX_SEMAPHORES) && !defined(HAVE_BROKEN_POSIX_SEMAPHORES) && \ - defined(HAVE_SEM_TIMEDWAIT)) + (defined(HAVE_SEM_TIMEDWAIT) || defined(HAVE_SEM_CLOCKWAIT))) # define USE_SEMAPHORES #else # undef USE_SEMAPHORES @@ -461,17 +461,34 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, timeout = _PyTime_FromNanoseconds(-1); } +#ifdef HAVE_SEM_CLOCKWAIT + struct timespec abs_timeout; + // Local scope for deadline + { + _PyTime_t deadline = _PyTime_GetMonotonicClock() + timeout; + _PyTime_AsTimespec_clamp(deadline, &abs_timeout); + } +#else _PyTime_t deadline = 0; - if (timeout > 0 && !intr_flag) { + if (timeout > 0 + && !intr_flag + ) + { deadline = _PyTime_GetMonotonicClock() + timeout; } +#endif while (1) { if (timeout > 0) { - _PyTime_t t = _PyTime_GetSystemClock() + timeout; +#ifdef HAVE_SEM_CLOCKWAIT + status = fix_status(sem_clockwait(thelock, CLOCK_MONOTONIC, + &abs_timeout)); +#else + _PyTime_t abs_timeout = _PyTime_GetSystemClock() + timeout; struct timespec ts; - _PyTime_AsTimespec_clamp(t, &ts); + _PyTime_AsTimespec_clamp(abs_timeout, &ts); status = fix_status(sem_timedwait(thelock, &ts)); +#endif } else if (timeout == 0) { status = fix_status(sem_trywait(thelock)); @@ -486,6 +503,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, break; } + // sem_clockwait() uses an absolute timeout, there is no need + // to recompute the relative timeout. +#ifndef HAVE_SEM_CLOCKWAIT if (timeout > 0) { /* wait interrupted by a signal (EINTR): recompute the timeout */ _PyTime_t timeout = deadline - _PyTime_GetMonotonicClock(); @@ -494,17 +514,24 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, break; } } +#endif } /* Don't check the status if we're stopping because of an interrupt. */ if (!(intr_flag && status == EINTR)) { if (timeout > 0) { - if (status != ETIMEDOUT) + if (status != ETIMEDOUT) { +#ifdef HAVE_SEM_CLOCKWAIT + CHECK_STATUS("sem_clockwait"); +#else CHECK_STATUS("sem_timedwait"); +#endif + } } else if (timeout == 0) { - if (status != EAGAIN) + if (status != EAGAIN) { CHECK_STATUS("sem_trywait"); + } } else { CHECK_STATUS("sem_wait"); diff --git a/configure b/configure index 4acf91f22107f..75e2e296f10b1 100755 --- a/configure +++ b/configure @@ -9764,7 +9764,7 @@ then BLDSHARED="$LDSHARED" fi ;; - Linux*|GNU*|QNX*|VxWorks*) + Linux*|GNU*|QNX*|VxWorks*|Haiku*) LDSHARED='$(CC) -shared' LDCXXSHARED='$(CXX) -shared';; FreeBSD*) @@ -9835,6 +9835,7 @@ then Linux-android*) ;; Linux*|GNU*) CCSHARED="-fPIC";; FreeBSD*|NetBSD*|OpenBSD*|DragonFly*) CCSHARED="-fPIC";; + Haiku*) CCSHARED="-fPIC";; OpenUNIX*|UnixWare*) if test "$GCC" = "yes" then CCSHARED="-fPIC" @@ -10562,6 +10563,48 @@ if test "x$ac_cv_lib_socket_socket" = xyes; then : fi # SVR4 sockets +# Haiku system library +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lnetwork" >&5 +$as_echo_n "checking for socket in -lnetwork... " >&6; } +if ${ac_cv_lib_network_socket+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-lnetwork $LIBS $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char socket (); +int +main () +{ +return socket (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_network_socket=yes +else + ac_cv_lib_network_socket=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_network_socket" >&5 +$as_echo "$ac_cv_lib_network_socket" >&6; } +if test "x$ac_cv_lib_network_socket" = xyes; then : + LIBS="-lnetwork $LIBS" +fi + + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-libs" >&5 $as_echo_n "checking for --with-libs... " >&6; } @@ -11774,7 +11817,7 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ @@ -13252,19 +13295,19 @@ fi done -for ac_func in clock_nanosleep +for ac_func in clock_getres do : - ac_fn_c_check_func "$LINENO" "clock_nanosleep" "ac_cv_func_clock_nanosleep" -if test "x$ac_cv_func_clock_nanosleep" = xyes; then : + ac_fn_c_check_func "$LINENO" "clock_getres" "ac_cv_func_clock_getres" +if test "x$ac_cv_func_clock_getres" = xyes; then : cat >>confdefs.h <<_ACEOF -#define HAVE_CLOCK_NANOSLEEP 1 +#define HAVE_CLOCK_GETRES 1 _ACEOF else - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_nanosleep in -lrt" >&5 -$as_echo_n "checking for clock_nanosleep in -lrt... " >&6; } -if ${ac_cv_lib_rt_clock_nanosleep+:} false; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_getres in -lrt" >&5 +$as_echo_n "checking for clock_getres in -lrt... " >&6; } +if ${ac_cv_lib_rt_clock_getres+:} false; then : $as_echo_n "(cached) " >&6 else ac_check_lib_save_LIBS=$LIBS @@ -13278,29 +13321,29 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext #ifdef __cplusplus extern "C" #endif -char clock_nanosleep (); +char clock_getres (); int main () { -return clock_nanosleep (); +return clock_getres (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_rt_clock_nanosleep=yes + ac_cv_lib_rt_clock_getres=yes else - ac_cv_lib_rt_clock_nanosleep=no + ac_cv_lib_rt_clock_getres=no fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext LIBS=$ac_check_lib_save_LIBS fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_nanosleep" >&5 -$as_echo "$ac_cv_lib_rt_clock_nanosleep" >&6; } -if test "x$ac_cv_lib_rt_clock_nanosleep" = xyes; then : +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_getres" >&5 +$as_echo "$ac_cv_lib_rt_clock_getres" >&6; } +if test "x$ac_cv_lib_rt_clock_getres" = xyes; then : - $as_echo "#define HAVE_CLOCK_NANOSLEEP 1" >>confdefs.h + $as_echo "#define HAVE_CLOCK_GETRES 1" >>confdefs.h fi @@ -13310,19 +13353,19 @@ fi done -for ac_func in nanosleep +for ac_func in clock_settime do : - ac_fn_c_check_func "$LINENO" "nanosleep" "ac_cv_func_nanosleep" -if test "x$ac_cv_func_nanosleep" = xyes; then : + ac_fn_c_check_func "$LINENO" "clock_settime" "ac_cv_func_clock_settime" +if test "x$ac_cv_func_clock_settime" = xyes; then : cat >>confdefs.h <<_ACEOF -#define HAVE_NANOSLEEP 1 +#define HAVE_CLOCK_SETTIME 1 _ACEOF else - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for nanosleep in -lrt" >&5 -$as_echo_n "checking for nanosleep in -lrt... " >&6; } -if ${ac_cv_lib_rt_nanosleep+:} false; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_settime in -lrt" >&5 +$as_echo_n "checking for clock_settime in -lrt... " >&6; } +if ${ac_cv_lib_rt_clock_settime+:} false; then : $as_echo_n "(cached) " >&6 else ac_check_lib_save_LIBS=$LIBS @@ -13336,29 +13379,29 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext #ifdef __cplusplus extern "C" #endif -char nanosleep (); +char clock_settime (); int main () { -return nanosleep (); +return clock_settime (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_rt_nanosleep=yes + ac_cv_lib_rt_clock_settime=yes else - ac_cv_lib_rt_nanosleep=no + ac_cv_lib_rt_clock_settime=no fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext LIBS=$ac_check_lib_save_LIBS fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_nanosleep" >&5 -$as_echo "$ac_cv_lib_rt_nanosleep" >&6; } -if test "x$ac_cv_lib_rt_nanosleep" = xyes; then : +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_settime" >&5 +$as_echo "$ac_cv_lib_rt_clock_settime" >&6; } +if test "x$ac_cv_lib_rt_clock_settime" = xyes; then : - $as_echo "#define HAVE_NANOSLEEP 1" >>confdefs.h + $as_echo "#define HAVE_CLOCK_SETTIME 1" >>confdefs.h fi @@ -13368,19 +13411,19 @@ fi done -for ac_func in clock_getres +for ac_func in clock_nanosleep do : - ac_fn_c_check_func "$LINENO" "clock_getres" "ac_cv_func_clock_getres" -if test "x$ac_cv_func_clock_getres" = xyes; then : + ac_fn_c_check_func "$LINENO" "clock_nanosleep" "ac_cv_func_clock_nanosleep" +if test "x$ac_cv_func_clock_nanosleep" = xyes; then : cat >>confdefs.h <<_ACEOF -#define HAVE_CLOCK_GETRES 1 +#define HAVE_CLOCK_NANOSLEEP 1 _ACEOF else - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_getres in -lrt" >&5 -$as_echo_n "checking for clock_getres in -lrt... " >&6; } -if ${ac_cv_lib_rt_clock_getres+:} false; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_nanosleep in -lrt" >&5 +$as_echo_n "checking for clock_nanosleep in -lrt... " >&6; } +if ${ac_cv_lib_rt_clock_nanosleep+:} false; then : $as_echo_n "(cached) " >&6 else ac_check_lib_save_LIBS=$LIBS @@ -13394,29 +13437,29 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext #ifdef __cplusplus extern "C" #endif -char clock_getres (); +char clock_nanosleep (); int main () { -return clock_getres (); +return clock_nanosleep (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_rt_clock_getres=yes + ac_cv_lib_rt_clock_nanosleep=yes else - ac_cv_lib_rt_clock_getres=no + ac_cv_lib_rt_clock_nanosleep=no fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext LIBS=$ac_check_lib_save_LIBS fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_getres" >&5 -$as_echo "$ac_cv_lib_rt_clock_getres" >&6; } -if test "x$ac_cv_lib_rt_clock_getres" = xyes; then : +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_nanosleep" >&5 +$as_echo "$ac_cv_lib_rt_clock_nanosleep" >&6; } +if test "x$ac_cv_lib_rt_clock_nanosleep" = xyes; then : - $as_echo "#define HAVE_CLOCK_GETRES 1" >>confdefs.h + $as_echo "#define HAVE_CLOCK_NANOSLEEP 1" >>confdefs.h fi @@ -13426,19 +13469,19 @@ fi done -for ac_func in clock_settime +for ac_func in nanosleep do : - ac_fn_c_check_func "$LINENO" "clock_settime" "ac_cv_func_clock_settime" -if test "x$ac_cv_func_clock_settime" = xyes; then : + ac_fn_c_check_func "$LINENO" "nanosleep" "ac_cv_func_nanosleep" +if test "x$ac_cv_func_nanosleep" = xyes; then : cat >>confdefs.h <<_ACEOF -#define HAVE_CLOCK_SETTIME 1 +#define HAVE_NANOSLEEP 1 _ACEOF else - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for clock_settime in -lrt" >&5 -$as_echo_n "checking for clock_settime in -lrt... " >&6; } -if ${ac_cv_lib_rt_clock_settime+:} false; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for nanosleep in -lrt" >&5 +$as_echo_n "checking for nanosleep in -lrt... " >&6; } +if ${ac_cv_lib_rt_nanosleep+:} false; then : $as_echo_n "(cached) " >&6 else ac_check_lib_save_LIBS=$LIBS @@ -13452,29 +13495,29 @@ cat confdefs.h - <<_ACEOF >conftest.$ac_ext #ifdef __cplusplus extern "C" #endif -char clock_settime (); +char nanosleep (); int main () { -return clock_settime (); +return nanosleep (); ; return 0; } _ACEOF if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_rt_clock_settime=yes + ac_cv_lib_rt_nanosleep=yes else - ac_cv_lib_rt_clock_settime=no + ac_cv_lib_rt_nanosleep=no fi rm -f core conftest.err conftest.$ac_objext \ conftest$ac_exeext conftest.$ac_ext LIBS=$ac_check_lib_save_LIBS fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_clock_settime" >&5 -$as_echo "$ac_cv_lib_rt_clock_settime" >&6; } -if test "x$ac_cv_lib_rt_clock_settime" = xyes; then : +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_rt_nanosleep" >&5 +$as_echo "$ac_cv_lib_rt_nanosleep" >&6; } +if test "x$ac_cv_lib_rt_nanosleep" = xyes; then : - $as_echo "#define HAVE_CLOCK_SETTIME 1" >>confdefs.h + $as_echo "#define HAVE_NANOSLEEP 1" >>confdefs.h fi diff --git a/configure.ac b/configure.ac index 48d86ef79199e..908dd28e7aaca 100644 --- a/configure.ac +++ b/configure.ac @@ -3744,7 +3744,7 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ diff --git a/pyconfig.h.in b/pyconfig.h.in index 23d7111b9f77e..862c083604ee4 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -136,15 +136,15 @@ /* Define to 1 if you have the `clock' function. */ #undef HAVE_CLOCK -/* Define to 1 if you have the `clock_nanosleep' function. */ -#undef HAVE_CLOCK_NANOSLEEP - /* Define to 1 if you have the `clock_getres' function. */ #undef HAVE_CLOCK_GETRES /* Define to 1 if you have the `clock_gettime' function. */ #undef HAVE_CLOCK_GETTIME +/* Define to 1 if you have the `clock_nanosleep' function. */ +#undef HAVE_CLOCK_NANOSLEEP + /* Define to 1 if you have the `clock_settime' function. */ #undef HAVE_CLOCK_SETTIME @@ -908,6 +908,9 @@ /* Define to 1 if you have the `sched_setscheduler' function. */ #undef HAVE_SCHED_SETSCHEDULER +/* Define to 1 if you have the `sem_clockwait' function. */ +#undef HAVE_SEM_CLOCKWAIT + /* Define to 1 if you have the `sem_getvalue' function. */ #undef HAVE_SEM_GETVALUE From webhook-mailer at python.org Fri Oct 1 03:56:37 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 01 Oct 2021 07:56:37 -0000 Subject: [Python-checkins] bpo-45310: Fix parrallel shared memory tests (GH-28661) Message-ID: https://github.com/python/cpython/commit/eb4495e8e275c83d691add116c4f2b74e73e3cc8 commit: eb4495e8e275c83d691add116c4f2b74e73e3cc8 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-01T10:56:32+03:00 summary: bpo-45310: Fix parrallel shared memory tests (GH-28661) Add a PID to names of POSIX shared memory objects to allow running multiprocessing tests (test_multiprocessing_fork, test_multiprocessing_spawn, etc) in parallel. files: M Lib/test/_test_multiprocessing.py diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 125e8906d8abc..9e0d18da50f6e 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3773,12 +3773,19 @@ def _attach_existing_shmem_then_write(shmem_name_or_obj, binary_data): local_sms.buf[:len(binary_data)] = binary_data local_sms.close() + def _new_shm_name(self, prefix): + # Add a PID to the name of a POSIX shared memory object to allow + # running multiprocessing tests (test_multiprocessing_fork, + # test_multiprocessing_spawn, etc) in parallel. + return prefix + str(os.getpid()) + def test_shared_memory_basics(self): - sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) + name_tsmb = self._new_shm_name('test01_tsmb') + sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512) self.addCleanup(sms.unlink) # Verify attributes are readable. - self.assertEqual(sms.name, 'test01_tsmb') + self.assertEqual(sms.name, name_tsmb) self.assertGreaterEqual(sms.size, 512) self.assertGreaterEqual(len(sms.buf), sms.size) @@ -3798,12 +3805,12 @@ def test_shared_memory_basics(self): self.assertEqual(sms.buf[0], 42) # Attach to existing shared memory segment. - also_sms = shared_memory.SharedMemory('test01_tsmb') + also_sms = shared_memory.SharedMemory(name_tsmb) self.assertEqual(also_sms.buf[0], 42) also_sms.close() # Attach to existing shared memory segment but specify a new size. - same_sms = shared_memory.SharedMemory('test01_tsmb', size=20*sms.size) + same_sms = shared_memory.SharedMemory(name_tsmb, size=20*sms.size) self.assertLess(same_sms.size, 20*sms.size) # Size was ignored. same_sms.close() @@ -3821,7 +3828,7 @@ def test_shared_memory_basics(self): 'multiprocessing.shared_memory._make_filename') as mock_make_filename: NAME_PREFIX = shared_memory._SHM_NAME_PREFIX - names = ['test01_fn', 'test02_fn'] + names = [self._new_shm_name('test01_fn'), self._new_shm_name('test02_fn')] # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary # because some POSIX compliant systems require name to start with / names = [NAME_PREFIX + name for name in names] @@ -3843,17 +3850,17 @@ def test_shared_memory_basics(self): # manages unlinking on its own and unlink() does nothing). # True release of shared memory segment does not necessarily # happen until process exits, depending on the OS platform. + name_dblunlink = self._new_shm_name('test01_dblunlink') + sms_uno = shared_memory.SharedMemory( + name_dblunlink, + create=True, + size=5000 + ) with self.assertRaises(FileNotFoundError): - sms_uno = shared_memory.SharedMemory( - 'test01_dblunlink', - create=True, - size=5000 - ) - try: self.assertGreaterEqual(sms_uno.size, 5000) - sms_duo = shared_memory.SharedMemory('test01_dblunlink') + sms_duo = shared_memory.SharedMemory(name_dblunlink) sms_duo.unlink() # First shm_unlink() call. sms_duo.close() sms_uno.close() @@ -3865,7 +3872,7 @@ def test_shared_memory_basics(self): # Attempting to create a new shared memory segment with a # name that is already in use triggers an exception. there_can_only_be_one_sms = shared_memory.SharedMemory( - 'test01_tsmb', + name_tsmb, create=True, size=512 ) @@ -3879,7 +3886,7 @@ def test_shared_memory_basics(self): # case of MacOS/darwin, requesting a smaller size is disallowed. class OptionalAttachSharedMemory(shared_memory.SharedMemory): _flags = os.O_CREAT | os.O_RDWR - ok_if_exists_sms = OptionalAttachSharedMemory('test01_tsmb') + ok_if_exists_sms = OptionalAttachSharedMemory(name_tsmb) self.assertEqual(ok_if_exists_sms.size, sms.size) ok_if_exists_sms.close() @@ -4084,10 +4091,11 @@ def test_shared_memory_ShareableList_basics(self): self.assertEqual(sl.count(b'adios'), 0) # Exercise creating a duplicate. - sl_copy = shared_memory.ShareableList(sl, name='test03_duplicate') + name_duplicate = self._new_shm_name('test03_duplicate') + sl_copy = shared_memory.ShareableList(sl, name=name_duplicate) try: self.assertNotEqual(sl.shm.name, sl_copy.shm.name) - self.assertEqual('test03_duplicate', sl_copy.shm.name) + self.assertEqual(name_duplicate, sl_copy.shm.name) self.assertEqual(list(sl), list(sl_copy)) self.assertEqual(sl.format, sl_copy.format) sl_copy[-1] = 77 From webhook-mailer at python.org Fri Oct 1 03:58:03 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 01 Oct 2021 07:58:03 -0000 Subject: [Python-checkins] Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)" (GH-28667) Message-ID: https://github.com/python/cpython/commit/2f205920127bd93eebed044cb1b61834764478ba commit: 2f205920127bd93eebed044cb1b61834764478ba branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-01T10:57:58+03:00 summary: Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615)" (GH-28650)" (GH-28667) This reverts commit b07fddd527efe67174ce6b0fdbe8dac390b16e4e. files: M Lib/test/test_datetime.py diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py index bdb9f02e5756a..7f9094fa7bd4e 100644 --- a/Lib/test/test_datetime.py +++ b/Lib/test/test_datetime.py @@ -1,59 +1,57 @@ import unittest import sys -from test.support import run_unittest from test.support.import_helper import import_fresh_module TESTS = 'test.datetimetester' -try: - pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], - blocked=['_datetime']) - fast_tests = import_fresh_module(TESTS, fresh=['datetime', - '_datetime', '_strptime']) -finally: - # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, - # XXX: but it does not, so we have to cleanup ourselves. - for modname in ['datetime', '_datetime', '_strptime']: - sys.modules.pop(modname, None) -test_modules = [pure_tests, fast_tests] -test_suffixes = ["_Pure", "_Fast"] -# XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might -# not believe this, but in spite of all the sys.modules trickery running a _Pure -# test last will leave a mix of pure and native datetime stuff lying around. -all_test_classes = [] +def load_tests(loader, tests, pattern): + try: + pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], + blocked=['_datetime']) + fast_tests = import_fresh_module(TESTS, fresh=['datetime', + '_datetime', '_strptime']) + finally: + # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, + # XXX: but it does not, so we have to cleanup ourselves. + for modname in ['datetime', '_datetime', '_strptime']: + sys.modules.pop(modname, None) -for module, suffix in zip(test_modules, test_suffixes): - test_classes = [] - for name, cls in module.__dict__.items(): - if not isinstance(cls, type): - continue - if issubclass(cls, unittest.TestCase): - test_classes.append(cls) - elif issubclass(cls, unittest.TestSuite): - suit = cls() - test_classes.extend(type(test) for test in suit) - test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) - for cls in test_classes: - cls.__name__ += suffix - cls.__qualname__ += suffix - @classmethod - def setUpClass(cls_, module=module): - cls_._save_sys_modules = sys.modules.copy() - sys.modules[TESTS] = module - sys.modules['datetime'] = module.datetime_module - sys.modules['_strptime'] = module._strptime - @classmethod - def tearDownClass(cls_): - sys.modules.clear() - sys.modules.update(cls_._save_sys_modules) - cls.setUpClass = setUpClass - cls.tearDownClass = tearDownClass - all_test_classes.extend(test_classes) + test_modules = [pure_tests, fast_tests] + test_suffixes = ["_Pure", "_Fast"] + # XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might + # not believe this, but in spite of all the sys.modules trickery running a _Pure + # test last will leave a mix of pure and native datetime stuff lying around. + for module, suffix in zip(test_modules, test_suffixes): + test_classes = [] + for name, cls in module.__dict__.items(): + if not isinstance(cls, type): + continue + if issubclass(cls, unittest.TestCase): + test_classes.append(cls) + elif issubclass(cls, unittest.TestSuite): + suit = cls() + test_classes.extend(type(test) for test in suit) + test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) + for cls in test_classes: + cls.__name__ += suffix + cls.__qualname__ += suffix + @classmethod + def setUpClass(cls_, module=module): + cls_._save_sys_modules = sys.modules.copy() + sys.modules[TESTS] = module + sys.modules['datetime'] = module.datetime_module + sys.modules['_strptime'] = module._strptime + @classmethod + def tearDownClass(cls_): + sys.modules.clear() + sys.modules.update(cls_._save_sys_modules) + cls.setUpClass = setUpClass + cls.tearDownClass = tearDownClass + tests.addTests(loader.loadTestsFromTestCase(cls)) + return tests -def test_main(): - run_unittest(*all_test_classes) if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 1 03:59:03 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 01 Oct 2021 07:59:03 -0000 Subject: [Python-checkins] Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)" (GH-28666) Message-ID: https://github.com/python/cpython/commit/e9d5cdda1ffa369550a634a3ec220db93433e0f4 commit: e9d5cdda1ffa369550a634a3ec220db93433e0f4 branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-01T10:58:59+03:00 summary: Revert "Revert "bpo-45229: Make datetime tests discoverable (GH-28615). (GH-28645)" (GH-28660)" (GH-28666) This reverts commit 2cf76cf4ccd177b8d6d2bf21b5462258ae87522d. files: M Lib/test/test_datetime.py diff --git a/Lib/test/test_datetime.py b/Lib/test/test_datetime.py index d659f369d54e4..c26dbe1063f41 100644 --- a/Lib/test/test_datetime.py +++ b/Lib/test/test_datetime.py @@ -1,57 +1,57 @@ import unittest import sys -from test.support import import_fresh_module, run_unittest +from test.support import import_fresh_module + TESTS = 'test.datetimetester' -try: - pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], - blocked=['_datetime']) - fast_tests = import_fresh_module(TESTS, fresh=['datetime', - '_datetime', '_strptime']) -finally: - # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, - # XXX: but it does not, so we have to cleanup ourselves. - for modname in ['datetime', '_datetime', '_strptime']: - sys.modules.pop(modname, None) -test_modules = [pure_tests, fast_tests] -test_suffixes = ["_Pure", "_Fast"] -# XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might -# not believe this, but in spite of all the sys.modules trickery running a _Pure -# test last will leave a mix of pure and native datetime stuff lying around. -all_test_classes = [] +def load_tests(loader, tests, pattern): + try: + pure_tests = import_fresh_module(TESTS, fresh=['datetime', '_strptime'], + blocked=['_datetime']) + fast_tests = import_fresh_module(TESTS, fresh=['datetime', + '_datetime', '_strptime']) + finally: + # XXX: import_fresh_module() is supposed to leave sys.module cache untouched, + # XXX: but it does not, so we have to cleanup ourselves. + for modname in ['datetime', '_datetime', '_strptime']: + sys.modules.pop(modname, None) -for module, suffix in zip(test_modules, test_suffixes): - test_classes = [] - for name, cls in module.__dict__.items(): - if not isinstance(cls, type): - continue - if issubclass(cls, unittest.TestCase): - test_classes.append(cls) - elif issubclass(cls, unittest.TestSuite): - suit = cls() - test_classes.extend(type(test) for test in suit) - test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) - for cls in test_classes: - cls.__name__ += suffix - cls.__qualname__ += suffix - @classmethod - def setUpClass(cls_, module=module): - cls_._save_sys_modules = sys.modules.copy() - sys.modules[TESTS] = module - sys.modules['datetime'] = module.datetime_module - sys.modules['_strptime'] = module._strptime - @classmethod - def tearDownClass(cls_): - sys.modules.clear() - sys.modules.update(cls_._save_sys_modules) - cls.setUpClass = setUpClass - cls.tearDownClass = tearDownClass - all_test_classes.extend(test_classes) + test_modules = [pure_tests, fast_tests] + test_suffixes = ["_Pure", "_Fast"] + # XXX(gb) First run all the _Pure tests, then all the _Fast tests. You might + # not believe this, but in spite of all the sys.modules trickery running a _Pure + # test last will leave a mix of pure and native datetime stuff lying around. + for module, suffix in zip(test_modules, test_suffixes): + test_classes = [] + for name, cls in module.__dict__.items(): + if not isinstance(cls, type): + continue + if issubclass(cls, unittest.TestCase): + test_classes.append(cls) + elif issubclass(cls, unittest.TestSuite): + suit = cls() + test_classes.extend(type(test) for test in suit) + test_classes = sorted(set(test_classes), key=lambda cls: cls.__qualname__) + for cls in test_classes: + cls.__name__ += suffix + cls.__qualname__ += suffix + @classmethod + def setUpClass(cls_, module=module): + cls_._save_sys_modules = sys.modules.copy() + sys.modules[TESTS] = module + sys.modules['datetime'] = module.datetime_module + sys.modules['_strptime'] = module._strptime + @classmethod + def tearDownClass(cls_): + sys.modules.clear() + sys.modules.update(cls_._save_sys_modules) + cls.setUpClass = setUpClass + cls.tearDownClass = tearDownClass + tests.addTests(loader.loadTestsFromTestCase(cls)) + return tests -def test_main(): - run_unittest(*all_test_classes) if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 1 05:38:32 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 01 Oct 2021 09:38:32 -0000 Subject: [Python-checkins] [3.9] bpo-45310: Fix parrallel shared memory tests (GH-28661) (GH-28670) Message-ID: https://github.com/python/cpython/commit/4d5d161d2aae41738d28e22bac5e1e08c01394bb commit: 4d5d161d2aae41738d28e22bac5e1e08c01394bb branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-01T12:38:23+03:00 summary: [3.9] bpo-45310: Fix parrallel shared memory tests (GH-28661) (GH-28670) Add a PID to names of POSIX shared memory objects to allow running multiprocessing tests (test_multiprocessing_fork, test_multiprocessing_spawn, etc) in parallel. (cherry picked from commit eb4495e8e275c83d691add116c4f2b74e73e3cc8) files: M Lib/test/_test_multiprocessing.py diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 6539f250c271a..3ae0cb976863d 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3769,13 +3769,20 @@ def _attach_existing_shmem_then_write(shmem_name_or_obj, binary_data): local_sms.buf[:len(binary_data)] = binary_data local_sms.close() + def _new_shm_name(self, prefix): + # Add a PID to the name of a POSIX shared memory object to allow + # running multiprocessing tests (test_multiprocessing_fork, + # test_multiprocessing_spawn, etc) in parallel. + return prefix + str(os.getpid()) + @unittest.skipIf(sys.platform == "win32", "test is broken on Windows") def test_shared_memory_basics(self): - sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) + name_tsmb = self._new_shm_name('test01_tsmb') + sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512) self.addCleanup(sms.unlink) # Verify attributes are readable. - self.assertEqual(sms.name, 'test01_tsmb') + self.assertEqual(sms.name, name_tsmb) self.assertGreaterEqual(sms.size, 512) self.assertGreaterEqual(len(sms.buf), sms.size) @@ -3784,12 +3791,12 @@ def test_shared_memory_basics(self): self.assertEqual(sms.buf[0], 42) # Attach to existing shared memory segment. - also_sms = shared_memory.SharedMemory('test01_tsmb') + also_sms = shared_memory.SharedMemory(name_tsmb) self.assertEqual(also_sms.buf[0], 42) also_sms.close() # Attach to existing shared memory segment but specify a new size. - same_sms = shared_memory.SharedMemory('test01_tsmb', size=20*sms.size) + same_sms = shared_memory.SharedMemory(name_tsmb, size=20*sms.size) self.assertLess(same_sms.size, 20*sms.size) # Size was ignored. same_sms.close() @@ -3807,7 +3814,7 @@ def test_shared_memory_basics(self): 'multiprocessing.shared_memory._make_filename') as mock_make_filename: NAME_PREFIX = shared_memory._SHM_NAME_PREFIX - names = ['test01_fn', 'test02_fn'] + names = [self._new_shm_name('test01_fn'), self._new_shm_name('test02_fn')] # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary # because some POSIX compliant systems require name to start with / names = [NAME_PREFIX + name for name in names] @@ -3829,17 +3836,17 @@ def test_shared_memory_basics(self): # manages unlinking on its own and unlink() does nothing). # True release of shared memory segment does not necessarily # happen until process exits, depending on the OS platform. + name_dblunlink = self._new_shm_name('test01_dblunlink') + sms_uno = shared_memory.SharedMemory( + name_dblunlink, + create=True, + size=5000 + ) with self.assertRaises(FileNotFoundError): - sms_uno = shared_memory.SharedMemory( - 'test01_dblunlink', - create=True, - size=5000 - ) - try: self.assertGreaterEqual(sms_uno.size, 5000) - sms_duo = shared_memory.SharedMemory('test01_dblunlink') + sms_duo = shared_memory.SharedMemory(name_dblunlink) sms_duo.unlink() # First shm_unlink() call. sms_duo.close() sms_uno.close() @@ -3851,7 +3858,7 @@ def test_shared_memory_basics(self): # Attempting to create a new shared memory segment with a # name that is already in use triggers an exception. there_can_only_be_one_sms = shared_memory.SharedMemory( - 'test01_tsmb', + name_tsmb, create=True, size=512 ) @@ -3865,7 +3872,7 @@ def test_shared_memory_basics(self): # case of MacOS/darwin, requesting a smaller size is disallowed. class OptionalAttachSharedMemory(shared_memory.SharedMemory): _flags = os.O_CREAT | os.O_RDWR - ok_if_exists_sms = OptionalAttachSharedMemory('test01_tsmb') + ok_if_exists_sms = OptionalAttachSharedMemory(name_tsmb) self.assertEqual(ok_if_exists_sms.size, sms.size) ok_if_exists_sms.close() @@ -4053,10 +4060,11 @@ def test_shared_memory_ShareableList_basics(self): self.assertEqual(sl.count(b'adios'), 0) # Exercise creating a duplicate. - sl_copy = shared_memory.ShareableList(sl, name='test03_duplicate') + name_duplicate = self._new_shm_name('test03_duplicate') + sl_copy = shared_memory.ShareableList(sl, name=name_duplicate) try: self.assertNotEqual(sl.shm.name, sl_copy.shm.name) - self.assertEqual('test03_duplicate', sl_copy.shm.name) + self.assertEqual(name_duplicate, sl_copy.shm.name) self.assertEqual(list(sl), list(sl_copy)) self.assertEqual(sl.format, sl_copy.format) sl_copy[-1] = 77 From webhook-mailer at python.org Fri Oct 1 05:41:25 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 09:41:25 -0000 Subject: [Python-checkins] bpo-45310: Fix parrallel shared memory tests (GH-28661) Message-ID: https://github.com/python/cpython/commit/4e6681d0cf9e7ce5621c34ff87055fa1f9786108 commit: 4e6681d0cf9e7ce5621c34ff87055fa1f9786108 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T02:41:20-07:00 summary: bpo-45310: Fix parrallel shared memory tests (GH-28661) Add a PID to names of POSIX shared memory objects to allow running multiprocessing tests (test_multiprocessing_fork, test_multiprocessing_spawn, etc) in parallel. (cherry picked from commit eb4495e8e275c83d691add116c4f2b74e73e3cc8) Co-authored-by: Serhiy Storchaka files: M Lib/test/_test_multiprocessing.py diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 125e8906d8abce..9e0d18da50f6ef 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3773,12 +3773,19 @@ def _attach_existing_shmem_then_write(shmem_name_or_obj, binary_data): local_sms.buf[:len(binary_data)] = binary_data local_sms.close() + def _new_shm_name(self, prefix): + # Add a PID to the name of a POSIX shared memory object to allow + # running multiprocessing tests (test_multiprocessing_fork, + # test_multiprocessing_spawn, etc) in parallel. + return prefix + str(os.getpid()) + def test_shared_memory_basics(self): - sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) + name_tsmb = self._new_shm_name('test01_tsmb') + sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512) self.addCleanup(sms.unlink) # Verify attributes are readable. - self.assertEqual(sms.name, 'test01_tsmb') + self.assertEqual(sms.name, name_tsmb) self.assertGreaterEqual(sms.size, 512) self.assertGreaterEqual(len(sms.buf), sms.size) @@ -3798,12 +3805,12 @@ def test_shared_memory_basics(self): self.assertEqual(sms.buf[0], 42) # Attach to existing shared memory segment. - also_sms = shared_memory.SharedMemory('test01_tsmb') + also_sms = shared_memory.SharedMemory(name_tsmb) self.assertEqual(also_sms.buf[0], 42) also_sms.close() # Attach to existing shared memory segment but specify a new size. - same_sms = shared_memory.SharedMemory('test01_tsmb', size=20*sms.size) + same_sms = shared_memory.SharedMemory(name_tsmb, size=20*sms.size) self.assertLess(same_sms.size, 20*sms.size) # Size was ignored. same_sms.close() @@ -3821,7 +3828,7 @@ def test_shared_memory_basics(self): 'multiprocessing.shared_memory._make_filename') as mock_make_filename: NAME_PREFIX = shared_memory._SHM_NAME_PREFIX - names = ['test01_fn', 'test02_fn'] + names = [self._new_shm_name('test01_fn'), self._new_shm_name('test02_fn')] # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary # because some POSIX compliant systems require name to start with / names = [NAME_PREFIX + name for name in names] @@ -3843,17 +3850,17 @@ def test_shared_memory_basics(self): # manages unlinking on its own and unlink() does nothing). # True release of shared memory segment does not necessarily # happen until process exits, depending on the OS platform. + name_dblunlink = self._new_shm_name('test01_dblunlink') + sms_uno = shared_memory.SharedMemory( + name_dblunlink, + create=True, + size=5000 + ) with self.assertRaises(FileNotFoundError): - sms_uno = shared_memory.SharedMemory( - 'test01_dblunlink', - create=True, - size=5000 - ) - try: self.assertGreaterEqual(sms_uno.size, 5000) - sms_duo = shared_memory.SharedMemory('test01_dblunlink') + sms_duo = shared_memory.SharedMemory(name_dblunlink) sms_duo.unlink() # First shm_unlink() call. sms_duo.close() sms_uno.close() @@ -3865,7 +3872,7 @@ def test_shared_memory_basics(self): # Attempting to create a new shared memory segment with a # name that is already in use triggers an exception. there_can_only_be_one_sms = shared_memory.SharedMemory( - 'test01_tsmb', + name_tsmb, create=True, size=512 ) @@ -3879,7 +3886,7 @@ def test_shared_memory_basics(self): # case of MacOS/darwin, requesting a smaller size is disallowed. class OptionalAttachSharedMemory(shared_memory.SharedMemory): _flags = os.O_CREAT | os.O_RDWR - ok_if_exists_sms = OptionalAttachSharedMemory('test01_tsmb') + ok_if_exists_sms = OptionalAttachSharedMemory(name_tsmb) self.assertEqual(ok_if_exists_sms.size, sms.size) ok_if_exists_sms.close() @@ -4084,10 +4091,11 @@ def test_shared_memory_ShareableList_basics(self): self.assertEqual(sl.count(b'adios'), 0) # Exercise creating a duplicate. - sl_copy = shared_memory.ShareableList(sl, name='test03_duplicate') + name_duplicate = self._new_shm_name('test03_duplicate') + sl_copy = shared_memory.ShareableList(sl, name=name_duplicate) try: self.assertNotEqual(sl.shm.name, sl_copy.shm.name) - self.assertEqual('test03_duplicate', sl_copy.shm.name) + self.assertEqual(name_duplicate, sl_copy.shm.name) self.assertEqual(list(sl), list(sl_copy)) self.assertEqual(sl.format, sl_copy.format) sl_copy[-1] = 77 From webhook-mailer at python.org Fri Oct 1 06:46:07 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 01 Oct 2021 10:46:07 -0000 Subject: [Python-checkins] bpo-45125: Improves pickling docs and tests for `shared_memory` (GH-28294) Message-ID: https://github.com/python/cpython/commit/746d648d47d12d16c2afedaeff626fc6aaaf6a46 commit: 746d648d47d12d16c2afedaeff626fc6aaaf6a46 branch: main author: Nikita Sobolev committer: serhiy-storchaka date: 2021-10-01T13:45:59+03:00 summary: bpo-45125: Improves pickling docs and tests for `shared_memory` (GH-28294) files: A Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst M Doc/library/multiprocessing.shared_memory.rst M Lib/test/_test_multiprocessing.py diff --git a/Doc/library/multiprocessing.shared_memory.rst b/Doc/library/multiprocessing.shared_memory.rst index cba576a29e2e2..2ba42b7e579a7 100644 --- a/Doc/library/multiprocessing.shared_memory.rst +++ b/Doc/library/multiprocessing.shared_memory.rst @@ -342,3 +342,30 @@ behind it: >>> c.shm.close() >>> c.shm.unlink() +The following examples demonstrates that ``ShareableList`` +(and underlying ``SharedMemory``) objects +can be pickled and unpickled if needed. +Note, that it will still be the same shared object. +This happens, because the deserialized object has +the same unique name and is just attached to an existing +object with the same name (if the object is still alive): + + >>> import pickle + >>> from multiprocessing import shared_memory + >>> sl = shared_memory.ShareableList(range(10)) + >>> list(sl) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> deserialized_sl = pickle.loads(pickle.dumps(sl)) + >>> list(deserialized_sl) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> sl[0] = -1 + >>> deserialized_sl[1] = -2 + >>> list(sl) + [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] + >>> list(deserialized_sl) + [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> sl.shm.close() + >>> sl.shm.unlink() diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 9e0d18da50f6e..3bc5b8f3d79b0 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3793,13 +3793,6 @@ def test_shared_memory_basics(self): self.assertIn(sms.name, str(sms)) self.assertIn(str(sms.size), str(sms)) - # Test pickling - sms.buf[0:6] = b'pickle' - pickled_sms = pickle.dumps(sms) - sms2 = pickle.loads(pickled_sms) - self.assertEqual(sms.name, sms2.name) - self.assertEqual(bytes(sms.buf[0:6]), bytes(sms2.buf[0:6]), b'pickle') - # Modify contents of shared memory segment through memoryview. sms.buf[0] = 42 self.assertEqual(sms.buf[0], 42) @@ -3898,6 +3891,29 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): sms.close() + def test_shared_memory_recreate(self): + # Test if shared memory segment is created properly, + # when _make_filename returns an existing shared memory segment name + with unittest.mock.patch( + 'multiprocessing.shared_memory._make_filename') as mock_make_filename: + + NAME_PREFIX = shared_memory._SHM_NAME_PREFIX + names = ['test01_fn', 'test02_fn'] + # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary + # because some POSIX compliant systems require name to start with / + names = [NAME_PREFIX + name for name in names] + + mock_make_filename.side_effect = names + shm1 = shared_memory.SharedMemory(create=True, size=1) + self.addCleanup(shm1.unlink) + self.assertEqual(shm1._name, names[0]) + + mock_make_filename.side_effect = names + shm2 = shared_memory.SharedMemory(create=True, size=1) + self.addCleanup(shm2.unlink) + self.assertEqual(shm2._name, names[1]) + + def test_invalid_shared_memory_cration(self): # Test creating a shared memory segment with negative size with self.assertRaises(ValueError): sms_invalid = shared_memory.SharedMemory(create=True, size=-1) @@ -3910,6 +3926,47 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): with self.assertRaises(ValueError): sms_invalid = shared_memory.SharedMemory(create=True) + def test_shared_memory_pickle_unpickle(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sms = shared_memory.SharedMemory(create=True, size=512) + self.addCleanup(sms.unlink) + sms.buf[0:6] = b'pickle' + + # Test pickling + pickled_sms = pickle.dumps(sms, protocol=proto) + + # Test unpickling + sms2 = pickle.loads(pickled_sms) + self.assertIsInstance(sms2, shared_memory.SharedMemory) + self.assertEqual(sms.name, sms2.name) + self.assertEqual(bytes(sms.buf[0:6]), b'pickle') + self.assertEqual(bytes(sms2.buf[0:6]), b'pickle') + + # Test that unpickled version is still the same SharedMemory + sms.buf[0:6] = b'newval' + self.assertEqual(bytes(sms.buf[0:6]), b'newval') + self.assertEqual(bytes(sms2.buf[0:6]), b'newval') + + sms2.buf[0:6] = b'oldval' + self.assertEqual(bytes(sms.buf[0:6]), b'oldval') + self.assertEqual(bytes(sms2.buf[0:6]), b'oldval') + + def test_shared_memory_pickle_unpickle_dead_object(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sms = shared_memory.SharedMemory(create=True, size=512) + sms.buf[0:6] = b'pickle' + pickled_sms = pickle.dumps(sms, protocol=proto) + + # Now, we are going to kill the original object. + # So, unpickled one won't be able to attach to it. + sms.close() + sms.unlink() + + with self.assertRaises(FileNotFoundError): + pickle.loads(pickled_sms) + def test_shared_memory_across_processes(self): # bpo-40135: don't define shared memory block's name in case of # the failure when we run multiprocessing tests in parallel. @@ -4127,29 +4184,45 @@ def test_shared_memory_ShareableList_basics(self): empty_sl.shm.unlink() def test_shared_memory_ShareableList_pickling(self): - sl = shared_memory.ShareableList(range(10)) - self.addCleanup(sl.shm.unlink) - - serialized_sl = pickle.dumps(sl) - deserialized_sl = pickle.loads(serialized_sl) - self.assertTrue( - isinstance(deserialized_sl, shared_memory.ShareableList) - ) - self.assertTrue(deserialized_sl[-1], 9) - self.assertFalse(sl is deserialized_sl) - deserialized_sl[4] = "changed" - self.assertEqual(sl[4], "changed") - - # Verify data is not being put into the pickled representation. - name = 'a' * len(sl.shm.name) - larger_sl = shared_memory.ShareableList(range(400)) - self.addCleanup(larger_sl.shm.unlink) - serialized_larger_sl = pickle.dumps(larger_sl) - self.assertTrue(len(serialized_sl) == len(serialized_larger_sl)) - larger_sl.shm.close() - - deserialized_sl.shm.close() - sl.shm.close() + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sl = shared_memory.ShareableList(range(10)) + self.addCleanup(sl.shm.unlink) + + serialized_sl = pickle.dumps(sl, protocol=proto) + deserialized_sl = pickle.loads(serialized_sl) + self.assertIsInstance( + deserialized_sl, shared_memory.ShareableList) + self.assertEqual(deserialized_sl[-1], 9) + self.assertIsNot(sl, deserialized_sl) + + deserialized_sl[4] = "changed" + self.assertEqual(sl[4], "changed") + sl[3] = "newvalue" + self.assertEqual(deserialized_sl[3], "newvalue") + + larger_sl = shared_memory.ShareableList(range(400)) + self.addCleanup(larger_sl.shm.unlink) + serialized_larger_sl = pickle.dumps(larger_sl, protocol=proto) + self.assertEqual(len(serialized_sl), len(serialized_larger_sl)) + larger_sl.shm.close() + + deserialized_sl.shm.close() + sl.shm.close() + + def test_shared_memory_ShareableList_pickling_dead_object(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sl = shared_memory.ShareableList(range(10)) + serialized_sl = pickle.dumps(sl, protocol=proto) + + # Now, we are going to kill the original object. + # So, unpickled one won't be able to attach to it. + sl.shm.close() + sl.shm.unlink() + + with self.assertRaises(FileNotFoundError): + pickle.loads(serialized_sl) def test_shared_memory_cleaned_after_process_termination(self): cmd = '''if 1: @@ -4202,7 +4275,7 @@ def test_shared_memory_cleaned_after_process_termination(self): "shared_memory objects to clean up at shutdown", err) # -# +# Test to verify that `Finalize` works. # class _TestFinalize(BaseTestCase): diff --git a/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst b/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst new file mode 100644 index 0000000000000..5dfbe0e5db463 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst @@ -0,0 +1,2 @@ +Improves pickling tests and docs of ``SharedMemory`` and ``SharableList`` +objects. From webhook-mailer at python.org Fri Oct 1 07:03:07 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 11:03:07 -0000 Subject: [Python-checkins] bpo-41710: Fix PY_TIMEOUT_MAX on Windows (GH-28673) Message-ID: https://github.com/python/cpython/commit/98d282700221234157159df4af76423d89490ad9 commit: 98d282700221234157159df4af76423d89490ad9 branch: main author: Victor Stinner committer: vstinner date: 2021-10-01T13:03:03+02:00 summary: bpo-41710: Fix PY_TIMEOUT_MAX on Windows (GH-28673) WaitForSingleObject() accepts timeout in milliseconds in the range [0; 0xFFFFFFFE] (DWORD type). INFINITE value (0xFFFFFFFF) means no timeout. 0xFFFFFFFE milliseconds is around 49.7 days. PY_TIMEOUT_MAX is (0xFFFFFFFE * 1000) milliseconds on Windows, around 49.7 days. Partially revert commit 37b8294d6295ca12553fd7c98778be71d24f4b24. files: D Misc/NEWS.d/next/Library/2021-09-30-08-57-50.bpo-41710.JMsPAW.rst M Include/pythread.h M Python/thread_nt.h diff --git a/Include/pythread.h b/Include/pythread.h index cf4cc9a7473f1..1a6092c4ad0be 100644 --- a/Include/pythread.h +++ b/Include/pythread.h @@ -61,11 +61,11 @@ PyAPI_FUNC(int) _PyThread_at_fork_reinit(PyThread_type_lock *lock); convert microseconds to nanoseconds. */ # define PY_TIMEOUT_MAX (LLONG_MAX / 1000) #elif defined (NT_THREADS) - /* In the NT API, the timeout is a DWORD and is expressed in milliseconds, - * a positive number between 0 and 0x7FFFFFFF (see WaitForSingleObject() - * documentation). */ -# if 0x7FFFFFFFLL * 1000 < LLONG_MAX -# define PY_TIMEOUT_MAX (0x7FFFFFFFLL * 1000) + // WaitForSingleObject() accepts timeout in milliseconds in the range + // [0; 0xFFFFFFFE] (DWORD type). INFINITE value (0xFFFFFFFF) means no + // timeout. 0xFFFFFFFE milliseconds is around 49.7 days. +# if 0xFFFFFFFELL * 1000 < LLONG_MAX +# define PY_TIMEOUT_MAX (0xFFFFFFFELL * 1000) # else # define PY_TIMEOUT_MAX LLONG_MAX # endif diff --git a/Misc/NEWS.d/next/Library/2021-09-30-08-57-50.bpo-41710.JMsPAW.rst b/Misc/NEWS.d/next/Library/2021-09-30-08-57-50.bpo-41710.JMsPAW.rst deleted file mode 100644 index 516214a619e8e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-30-08-57-50.bpo-41710.JMsPAW.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix :data:`_thread.TIMEOUT_MAX` value on Windows: the maximum timeout is -0x7FFFFFFF milliseconds (around 24.9 days), not 0xFFFFFFFF milliseconds (around -49.7 days). diff --git a/Python/thread_nt.h b/Python/thread_nt.h index 0beb3d3af2fd3..26f054ff0ce49 100644 --- a/Python/thread_nt.h +++ b/Python/thread_nt.h @@ -292,9 +292,10 @@ PyThread_free_lock(PyThread_type_lock aLock) FreeNonRecursiveMutex(aLock) ; } -// WaitForSingleObject() documentation: "The time-out value needs to be a -// positive number between 0 and 0x7FFFFFFF." INFINITE is equal to 0xFFFFFFFF. -const DWORD TIMEOUT_MS_MAX = 0x7FFFFFFF; +// WaitForSingleObject() accepts timeout in milliseconds in the range +// [0; 0xFFFFFFFE] (DWORD type). INFINITE value (0xFFFFFFFF) means no +// timeout. 0xFFFFFFFE milliseconds is around 49.7 days. +const DWORD TIMEOUT_MS_MAX = 0xFFFFFFFE; /* * Return 1 on success if the lock was acquired @@ -322,12 +323,11 @@ PyThread_acquire_lock_timed(PyThread_type_lock aLock, // overflow to the caller, so clamp the timeout to // [0, TIMEOUT_MS_MAX] milliseconds. // - // TIMEOUT_MS_MAX milliseconds is around 24.9 days. - // // _thread.Lock.acquire() and _thread.RLock.acquire() raise an // OverflowError if microseconds is greater than PY_TIMEOUT_MAX. milliseconds = TIMEOUT_MS_MAX; } + assert(milliseconds != INFINITE); } else { milliseconds = INFINITE; From webhook-mailer at python.org Fri Oct 1 07:29:14 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 11:29:14 -0000 Subject: [Python-checkins] bpo-41710: gc_collect_main() uses _PyTime_GetPerfCounter() (GH-28676) Message-ID: https://github.com/python/cpython/commit/54957f16a63ecb6b15f77b01fa7c55ada892604a commit: 54957f16a63ecb6b15f77b01fa7c55ada892604a branch: main author: Victor Stinner committer: vstinner date: 2021-10-01T13:29:00+02:00 summary: bpo-41710: gc_collect_main() uses _PyTime_GetPerfCounter() (GH-28676) If the DEBUG_STATS debug flag is set, gc_collect_main() now uses _PyTime_GetPerfCounter() instead of _PyTime_GetMonotonicClock() to measure the elapsed time. On Windows, _PyTime_GetMonotonicClock() only has a resolution of 15.6 ms, whereas _PyTime_GetPerfCounter() is closer to a resolution of 100 ns. files: M Modules/gcmodule.c diff --git a/Modules/gcmodule.c b/Modules/gcmodule.c index 2592c397cf2f7..7d1a45bcaeabf 100644 --- a/Modules/gcmodule.c +++ b/Modules/gcmodule.c @@ -1211,7 +1211,7 @@ gc_collect_main(PyThreadState *tstate, int generation, if (gcstate->debug & DEBUG_STATS) { PySys_WriteStderr("gc: collecting generation %d...\n", generation); show_stats_each_generations(gcstate); - t1 = _PyTime_GetMonotonicClock(); + t1 = _PyTime_GetPerfCounter(); } if (PyDTrace_GC_START_ENABLED()) @@ -1307,7 +1307,7 @@ gc_collect_main(PyThreadState *tstate, int generation, debug_cycle("uncollectable", FROM_GC(gc)); } if (gcstate->debug & DEBUG_STATS) { - double d = _PyTime_AsSecondsDouble(_PyTime_GetMonotonicClock() - t1); + double d = _PyTime_AsSecondsDouble(_PyTime_GetPerfCounter() - t1); PySys_WriteStderr( "gc: done, %zd unreachable, %zd uncollectable, %.4fs elapsed\n", n+m, n, d); From webhook-mailer at python.org Fri Oct 1 07:29:30 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 11:29:30 -0000 Subject: [Python-checkins] bpo-41710: Add private _PyDeadline_Get() function (GH-28674) Message-ID: https://github.com/python/cpython/commit/833fdf126c8fe77fd17e8a8ffbc5c571b3bf64bd commit: 833fdf126c8fe77fd17e8a8ffbc5c571b3bf64bd branch: main author: Victor Stinner committer: vstinner date: 2021-10-01T13:29:25+02:00 summary: bpo-41710: Add private _PyDeadline_Get() function (GH-28674) Add a private C API for deadlines: add _PyDeadline_Init() and _PyDeadline_Get() functions. * Add _PyTime_Add() and _PyTime_Mul() functions which compute t1+t2 and t1*t2 and clamp the result on overflow. * _PyTime_MulDiv() now uses _PyTime_Add() and _PyTime_Mul(). files: M Include/cpython/pytime.h M Modules/_queuemodule.c M Modules/_ssl.c M Modules/_threadmodule.c M Modules/clinic/_queuemodule.c.h M Modules/selectmodule.c M Modules/signalmodule.c M Modules/socketmodule.c M Python/pytime.c M Python/thread_nt.h M Python/thread_pthread.h diff --git a/Include/cpython/pytime.h b/Include/cpython/pytime.h index 5440720f1ba71..f32148aa8448d 100644 --- a/Include/cpython/pytime.h +++ b/Include/cpython/pytime.h @@ -27,6 +27,8 @@ // // Some functions clamp the result in the range [_PyTime_MIN; _PyTime_MAX], so // the caller doesn't have to handle errors and doesn't need to hold the GIL. +// For example, _PyTime_Add(t1, t2) computes t1+t2 and clamp the result on +// overflow. // // Clocks: // @@ -215,7 +217,12 @@ PyAPI_FUNC(int) _PyTime_AsTimespec(_PyTime_t t, struct timespec *ts); PyAPI_FUNC(void) _PyTime_AsTimespec_clamp(_PyTime_t t, struct timespec *ts); #endif + +// Compute t1 + t2. Clamp to [_PyTime_MIN; _PyTime_MAX] on overflow. +PyAPI_FUNC(_PyTime_t) _PyTime_Add(_PyTime_t t1, _PyTime_t t2); + /* Compute ticks * mul / div. + Clamp to [_PyTime_MIN; _PyTime_MAX] on overflow. The caller must ensure that ((div - 1) * mul) cannot overflow. */ PyAPI_FUNC(_PyTime_t) _PyTime_MulDiv(_PyTime_t ticks, _PyTime_t mul, @@ -299,6 +306,15 @@ PyAPI_FUNC(int) _PyTime_GetPerfCounterWithInfo( _PyTime_t *t, _Py_clock_info_t *info); + +// Create a deadline. +// Pseudo code: _PyTime_GetMonotonicClock() + timeout. +PyAPI_FUNC(_PyTime_t) _PyDeadline_Init(_PyTime_t timeout); + +// Get remaining time from a deadline. +// Pseudo code: deadline - _PyTime_GetMonotonicClock(). +PyAPI_FUNC(_PyTime_t) _PyDeadline_Get(_PyTime_t deadline); + #ifdef __cplusplus } #endif diff --git a/Modules/_queuemodule.c b/Modules/_queuemodule.c index b6769e6b7764e..eb61349b76581 100644 --- a/Modules/_queuemodule.c +++ b/Modules/_queuemodule.c @@ -182,7 +182,7 @@ _queue.SimpleQueue.get cls: defining_class / block: bool = True - timeout: object = None + timeout as timeout_obj: object = None Remove and return an item from the queue. @@ -198,11 +198,11 @@ in that case). static PyObject * _queue_SimpleQueue_get_impl(simplequeueobject *self, PyTypeObject *cls, - int block, PyObject *timeout) -/*[clinic end generated code: output=1969aefa7db63666 input=5fc4d56b9a54757e]*/ + int block, PyObject *timeout_obj) +/*[clinic end generated code: output=5c2cca914cd1e55b input=5b4047bfbc645ec1]*/ { _PyTime_t endtime = 0; - _PyTime_t timeout_val; + _PyTime_t timeout; PyObject *item; PyLockStatus r; PY_TIMEOUT_T microseconds; @@ -211,24 +211,25 @@ _queue_SimpleQueue_get_impl(simplequeueobject *self, PyTypeObject *cls, /* Non-blocking */ microseconds = 0; } - else if (timeout != Py_None) { + else if (timeout_obj != Py_None) { /* With timeout */ - if (_PyTime_FromSecondsObject(&timeout_val, - timeout, _PyTime_ROUND_CEILING) < 0) + if (_PyTime_FromSecondsObject(&timeout, + timeout_obj, _PyTime_ROUND_CEILING) < 0) { return NULL; - if (timeout_val < 0) { + } + if (timeout < 0) { PyErr_SetString(PyExc_ValueError, "'timeout' must be a non-negative number"); return NULL; } - microseconds = _PyTime_AsMicroseconds(timeout_val, + microseconds = _PyTime_AsMicroseconds(timeout, _PyTime_ROUND_CEILING); if (microseconds > PY_TIMEOUT_MAX) { PyErr_SetString(PyExc_OverflowError, "timeout value is too large"); return NULL; } - endtime = _PyTime_GetMonotonicClock() + timeout_val; + endtime = _PyDeadline_Init(timeout); } else { /* Infinitely blocking */ @@ -247,6 +248,7 @@ _queue_SimpleQueue_get_impl(simplequeueobject *self, PyTypeObject *cls, r = PyThread_acquire_lock_timed(self->lock, microseconds, 1); Py_END_ALLOW_THREADS } + if (r == PY_LOCK_INTR && Py_MakePendingCalls() < 0) { return NULL; } @@ -258,12 +260,15 @@ _queue_SimpleQueue_get_impl(simplequeueobject *self, PyTypeObject *cls, return NULL; } self->locked = 1; + /* Adjust timeout for next iteration (if any) */ - if (endtime > 0) { - timeout_val = endtime - _PyTime_GetMonotonicClock(); - microseconds = _PyTime_AsMicroseconds(timeout_val, _PyTime_ROUND_CEILING); + if (microseconds > 0) { + timeout = _PyDeadline_Get(endtime); + microseconds = _PyTime_AsMicroseconds(timeout, + _PyTime_ROUND_CEILING); } } + /* BEGIN GIL-protected critical section */ assert(self->lst_pos < PyList_GET_SIZE(self->lst)); item = simplequeue_pop_item(self); diff --git a/Modules/_ssl.c b/Modules/_ssl.c index 411314f6d0f01..fedb35b5778b3 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -949,8 +949,9 @@ _ssl__SSLSocket_do_handshake_impl(PySSLSocket *self) timeout = GET_SOCKET_TIMEOUT(sock); has_timeout = (timeout > 0); - if (has_timeout) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (has_timeout) { + deadline = _PyDeadline_Init(timeout); + } /* Actually negotiate SSL connection */ /* XXX If SSL_do_handshake() returns 0, it's also a failure. */ @@ -965,7 +966,7 @@ _ssl__SSLSocket_do_handshake_impl(PySSLSocket *self) goto error; if (has_timeout) - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (err.ssl == SSL_ERROR_WANT_READ) { sockstate = PySSL_select(sock, 0, timeout); @@ -2326,8 +2327,9 @@ _ssl__SSLSocket_write_impl(PySSLSocket *self, Py_buffer *b) timeout = GET_SOCKET_TIMEOUT(sock); has_timeout = (timeout > 0); - if (has_timeout) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (has_timeout) { + deadline = _PyDeadline_Init(timeout); + } sockstate = PySSL_select(sock, 1, timeout); if (sockstate == SOCKET_HAS_TIMED_OUT) { @@ -2354,8 +2356,9 @@ _ssl__SSLSocket_write_impl(PySSLSocket *self, Py_buffer *b) if (PyErr_CheckSignals()) goto error; - if (has_timeout) - timeout = deadline - _PyTime_GetMonotonicClock(); + if (has_timeout) { + timeout = _PyDeadline_Get(deadline); + } if (err.ssl == SSL_ERROR_WANT_READ) { sockstate = PySSL_select(sock, 0, timeout); @@ -2494,7 +2497,7 @@ _ssl__SSLSocket_read_impl(PySSLSocket *self, Py_ssize_t len, timeout = GET_SOCKET_TIMEOUT(sock); has_timeout = (timeout > 0); if (has_timeout) - deadline = _PyTime_GetMonotonicClock() + timeout; + deadline = _PyDeadline_Init(timeout); do { PySSL_BEGIN_ALLOW_THREADS @@ -2506,8 +2509,9 @@ _ssl__SSLSocket_read_impl(PySSLSocket *self, Py_ssize_t len, if (PyErr_CheckSignals()) goto error; - if (has_timeout) - timeout = deadline - _PyTime_GetMonotonicClock(); + if (has_timeout) { + timeout = _PyDeadline_Get(deadline); + } if (err.ssl == SSL_ERROR_WANT_READ) { sockstate = PySSL_select(sock, 0, timeout); @@ -2592,8 +2596,9 @@ _ssl__SSLSocket_shutdown_impl(PySSLSocket *self) timeout = GET_SOCKET_TIMEOUT(sock); has_timeout = (timeout > 0); - if (has_timeout) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (has_timeout) { + deadline = _PyDeadline_Init(timeout); + } while (1) { PySSL_BEGIN_ALLOW_THREADS @@ -2626,8 +2631,9 @@ _ssl__SSLSocket_shutdown_impl(PySSLSocket *self) continue; } - if (has_timeout) - timeout = deadline - _PyTime_GetMonotonicClock(); + if (has_timeout) { + timeout = _PyDeadline_Get(deadline); + } /* Possibly retry shutdown until timeout or failure */ if (err.ssl == SSL_ERROR_WANT_READ) diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index fa7e6d0e09d18..543d82d0b9382 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -84,13 +84,12 @@ lock_dealloc(lockobject *self) static PyLockStatus acquire_timed(PyThread_type_lock lock, _PyTime_t timeout) { - PyLockStatus r; _PyTime_t endtime = 0; - if (timeout > 0) { - endtime = _PyTime_GetMonotonicClock() + timeout; + endtime = _PyDeadline_Init(timeout); } + PyLockStatus r; do { _PyTime_t microseconds; microseconds = _PyTime_AsMicroseconds(timeout, _PyTime_ROUND_CEILING); @@ -114,7 +113,7 @@ acquire_timed(PyThread_type_lock lock, _PyTime_t timeout) /* If we're using a timeout, recompute the timeout after processing * signals, since those can take time. */ if (timeout > 0) { - timeout = endtime - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(endtime); /* Check for negative values, since those mean block forever. */ diff --git a/Modules/clinic/_queuemodule.c.h b/Modules/clinic/_queuemodule.c.h index d9522562359e6..22d2e992b6398 100644 --- a/Modules/clinic/_queuemodule.c.h +++ b/Modules/clinic/_queuemodule.c.h @@ -139,7 +139,7 @@ PyDoc_STRVAR(_queue_SimpleQueue_get__doc__, static PyObject * _queue_SimpleQueue_get_impl(simplequeueobject *self, PyTypeObject *cls, - int block, PyObject *timeout); + int block, PyObject *timeout_obj); static PyObject * _queue_SimpleQueue_get(simplequeueobject *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) @@ -148,13 +148,13 @@ _queue_SimpleQueue_get(simplequeueobject *self, PyTypeObject *cls, PyObject *con static const char * const _keywords[] = {"block", "timeout", NULL}; static _PyArg_Parser _parser = {"|pO:get", _keywords, 0}; int block = 1; - PyObject *timeout = Py_None; + PyObject *timeout_obj = Py_None; if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, - &block, &timeout)) { + &block, &timeout_obj)) { goto exit; } - return_value = _queue_SimpleQueue_get_impl(self, cls, block, timeout); + return_value = _queue_SimpleQueue_get_impl(self, cls, block, timeout_obj); exit: return return_value; @@ -248,4 +248,4 @@ _queue_SimpleQueue_qsize(simplequeueobject *self, PyObject *Py_UNUSED(ignored)) exit: return return_value; } -/*[clinic end generated code: output=96cc57168d72aab1 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=acfaf0191d8935db input=a9049054013a1b77]*/ diff --git a/Modules/selectmodule.c b/Modules/selectmodule.c index b71b2c4f6c037..ff1c028d0d672 100644 --- a/Modules/selectmodule.c +++ b/Modules/selectmodule.c @@ -318,8 +318,9 @@ select_select_impl(PyObject *module, PyObject *rlist, PyObject *wlist, if (omax > max) max = omax; if (emax > max) max = emax; - if (tvp) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (tvp) { + deadline = _PyDeadline_Init(timeout); + } do { Py_BEGIN_ALLOW_THREADS @@ -335,7 +336,7 @@ select_select_impl(PyObject *module, PyObject *rlist, PyObject *wlist, goto finally; if (tvp) { - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (timeout < 0) { /* bpo-35310: lists were unmodified -- clear them explicitly */ FD_ZERO(&ifdset); @@ -599,7 +600,7 @@ select_poll_poll_impl(pollObject *self, PyObject *timeout_obj) } if (timeout >= 0) { - deadline = _PyTime_GetMonotonicClock() + timeout; + deadline = _PyDeadline_Init(timeout); } } @@ -646,7 +647,7 @@ select_poll_poll_impl(pollObject *self, PyObject *timeout_obj) } if (timeout >= 0) { - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (timeout < 0) { poll_result = 0; break; @@ -938,8 +939,9 @@ select_devpoll_poll_impl(devpollObject *self, PyObject *timeout_obj) dvp.dp_nfds = self->max_n_fds; dvp.dp_timeout = (int)ms; - if (timeout >= 0) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (timeout >= 0) { + deadline = _PyDeadline_Init(timeout); + } do { /* call devpoll() */ @@ -956,7 +958,7 @@ select_devpoll_poll_impl(devpollObject *self, PyObject *timeout_obj) return NULL; if (timeout >= 0) { - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (timeout < 0) { poll_result = 0; break; @@ -1550,7 +1552,7 @@ select_epoll_poll_impl(pyEpoll_Object *self, PyObject *timeout_obj, } if (timeout >= 0) { - deadline = _PyTime_GetMonotonicClock() + timeout; + deadline = _PyDeadline_Init(timeout); } } @@ -1584,7 +1586,7 @@ select_epoll_poll_impl(pyEpoll_Object *self, PyObject *timeout_obj, goto error; if (timeout >= 0) { - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (timeout < 0) { nfds = 0; break; @@ -2172,8 +2174,9 @@ select_kqueue_control_impl(kqueue_queue_Object *self, PyObject *changelist, } } - if (ptimeoutspec) - deadline = _PyTime_GetMonotonicClock() + timeout; + if (ptimeoutspec) { + deadline = _PyDeadline_Init(timeout); + } do { Py_BEGIN_ALLOW_THREADS @@ -2190,7 +2193,7 @@ select_kqueue_control_impl(kqueue_queue_Object *self, PyObject *changelist, goto error; if (ptimeoutspec) { - timeout = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); if (timeout < 0) { gotevents = 0; break; diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index fc58cfd2084c7..8b7ef2cc688fe 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1221,11 +1221,7 @@ signal_sigtimedwait_impl(PyObject *module, sigset_t sigset, PyObject *timeout_obj) /*[clinic end generated code: output=59c8971e8ae18a64 input=87fd39237cf0b7ba]*/ { - struct timespec ts; - siginfo_t si; - int res; - _PyTime_t timeout, deadline, monotonic; - + _PyTime_t timeout; if (_PyTime_FromSecondsObject(&timeout, timeout_obj, _PyTime_ROUND_CEILING) < 0) return NULL; @@ -1235,12 +1231,16 @@ signal_sigtimedwait_impl(PyObject *module, sigset_t sigset, return NULL; } - deadline = _PyTime_GetMonotonicClock() + timeout; + _PyTime_t deadline = _PyDeadline_Init(timeout); + siginfo_t si; do { - if (_PyTime_AsTimespec(timeout, &ts) < 0) + struct timespec ts; + if (_PyTime_AsTimespec(timeout, &ts) < 0) { return NULL; + } + int res; Py_BEGIN_ALLOW_THREADS res = sigtimedwait(&sigset, &si, &ts); Py_END_ALLOW_THREADS @@ -1259,10 +1259,10 @@ signal_sigtimedwait_impl(PyObject *module, sigset_t sigset, if (PyErr_CheckSignals()) return NULL; - monotonic = _PyTime_GetMonotonicClock(); - timeout = deadline - monotonic; - if (timeout < 0) + timeout = _PyDeadline_Get(deadline); + if (timeout < 0) { break; + } } while (1); return fill_siginfo(&si); diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index f474869c94dc2..98212274b46a9 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -840,18 +840,20 @@ sock_call_ex(PySocketSockObject *s, if (deadline_initialized) { /* recompute the timeout */ - interval = deadline - _PyTime_GetMonotonicClock(); + interval = _PyDeadline_Get(deadline); } else { deadline_initialized = 1; - deadline = _PyTime_GetMonotonicClock() + timeout; + deadline = _PyDeadline_Init(timeout); interval = timeout; } - if (interval >= 0) + if (interval >= 0) { res = internal_select(s, writing, interval, connect); - else + } + else { res = 1; + } } else { res = internal_select(s, writing, timeout, connect); @@ -4176,7 +4178,7 @@ sock_sendall(PySocketSockObject *s, PyObject *args) Py_buffer pbuf; struct sock_send ctx; int has_timeout = (s->sock_timeout > 0); - _PyTime_t interval = s->sock_timeout; + _PyTime_t timeout = s->sock_timeout; _PyTime_t deadline = 0; int deadline_initialized = 0; PyObject *res = NULL; @@ -4195,14 +4197,14 @@ sock_sendall(PySocketSockObject *s, PyObject *args) if (has_timeout) { if (deadline_initialized) { /* recompute the timeout */ - interval = deadline - _PyTime_GetMonotonicClock(); + timeout = _PyDeadline_Get(deadline); } else { deadline_initialized = 1; - deadline = _PyTime_GetMonotonicClock() + s->sock_timeout; + deadline = _PyDeadline_Init(timeout); } - if (interval <= 0) { + if (timeout <= 0) { PyErr_SetString(PyExc_TimeoutError, "timed out"); goto done; } @@ -4211,7 +4213,7 @@ sock_sendall(PySocketSockObject *s, PyObject *args) ctx.buf = buf; ctx.len = len; ctx.flags = flags; - if (sock_call_ex(s, 1, sock_send_impl, &ctx, 0, NULL, interval) < 0) + if (sock_call_ex(s, 1, sock_send_impl, &ctx, 0, NULL, timeout) < 0) goto done; n = ctx.result; assert(n >= 0); diff --git a/Python/pytime.c b/Python/pytime.c index 7fd03ea576d27..8865638e91c23 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -73,30 +73,43 @@ pytime_as_nanoseconds(_PyTime_t t) } -// Compute t + t2. Clamp to [_PyTime_MIN; _PyTime_MAX] on overflow. +// Compute t1 + t2. Clamp to [_PyTime_MIN; _PyTime_MAX] on overflow. static inline int -pytime_add(_PyTime_t *t, _PyTime_t t2) +pytime_add(_PyTime_t *t1, _PyTime_t t2) { - if (t2 > 0 && *t > _PyTime_MAX - t2) { - *t = _PyTime_MAX; + if (t2 > 0 && *t1 > _PyTime_MAX - t2) { + *t1 = _PyTime_MAX; return -1; } - else if (t2 < 0 && *t < _PyTime_MIN - t2) { - *t = _PyTime_MIN; + else if (t2 < 0 && *t1 < _PyTime_MIN - t2) { + *t1 = _PyTime_MIN; return -1; } else { - *t += t2; + *t1 += t2; return 0; } } +_PyTime_t +_PyTime_Add(_PyTime_t t1, _PyTime_t t2) +{ + (void)pytime_add(&t1, t2); + return t1; +} + + static inline int -_PyTime_check_mul_overflow(_PyTime_t a, _PyTime_t b) +pytime_mul_check_overflow(_PyTime_t a, _PyTime_t b) { - assert(b > 0); - return ((a < _PyTime_MIN / b) || (_PyTime_MAX / b < a)); + if (b != 0) { + assert(b > 0); + return ((a < _PyTime_MIN / b) || (_PyTime_MAX / b < a)); + } + else { + return 0; + } } @@ -104,8 +117,8 @@ _PyTime_check_mul_overflow(_PyTime_t a, _PyTime_t b) static inline int pytime_mul(_PyTime_t *t, _PyTime_t k) { - assert(k > 0); - if (_PyTime_check_mul_overflow(*t, k)) { + assert(k >= 0); + if (pytime_mul_check_overflow(*t, k)) { *t = (*t >= 0) ? _PyTime_MAX : _PyTime_MIN; return -1; } @@ -116,21 +129,31 @@ pytime_mul(_PyTime_t *t, _PyTime_t k) } +// Compute t * k. Clamp to [_PyTime_MIN; _PyTime_MAX] on overflow. +static inline _PyTime_t +_PyTime_Mul(_PyTime_t t, _PyTime_t k) +{ + (void)pytime_mul(&t, k); + return t; +} + + + + _PyTime_t _PyTime_MulDiv(_PyTime_t ticks, _PyTime_t mul, _PyTime_t div) { - _PyTime_t intpart, remaining; - /* Compute (ticks * mul / div) in two parts to prevent integer overflow: - compute integer part, and then the remaining part. + /* Compute (ticks * mul / div) in two parts to reduce the risk of integer + overflow: compute the integer part, and then the remaining part. (ticks * mul) / div == (ticks / div) * mul + (ticks % div) * mul / div - - The caller must ensure that "(div - 1) * mul" cannot overflow. */ + */ + _PyTime_t intpart, remaining; intpart = ticks / div; ticks %= div; - remaining = ticks * mul; - remaining /= div; - return intpart * mul + remaining; + remaining = _PyTime_Mul(ticks, mul) / div; + // intpart * mul + remaining + return _PyTime_Add(_PyTime_Mul(intpart, mul), remaining); } @@ -505,7 +528,6 @@ pytime_from_object(_PyTime_t *tp, PyObject *obj, _PyTime_round_t round, return pytime_from_double(tp, d, round, unit_to_ns); } else { - Py_BUILD_ASSERT(sizeof(long long) <= sizeof(_PyTime_t)); long long sec = PyLong_AsLongLong(obj); if (sec == -1 && PyErr_Occurred()) { if (PyErr_ExceptionMatches(PyExc_OverflowError)) { @@ -514,11 +536,12 @@ pytime_from_object(_PyTime_t *tp, PyObject *obj, _PyTime_round_t round, return -1; } - if (_PyTime_check_mul_overflow(sec, unit_to_ns)) { + Py_BUILD_ASSERT(sizeof(long long) <= sizeof(_PyTime_t)); + _PyTime_t ns = (_PyTime_t)sec; + if (pytime_mul(&ns, unit_to_ns) < 0) { pytime_overflow(); return -1; } - _PyTime_t ns = sec * unit_to_ns; *tp = pytime_from_nanoseconds(ns); return 0; @@ -1292,3 +1315,19 @@ _PyTime_gmtime(time_t t, struct tm *tm) return 0; #endif /* MS_WINDOWS */ } + + +_PyTime_t +_PyDeadline_Init(_PyTime_t timeout) +{ + _PyTime_t now = _PyTime_GetMonotonicClock(); + return _PyTime_Add(now, timeout); +} + + +_PyTime_t +_PyDeadline_Get(_PyTime_t deadline) +{ + _PyTime_t now = _PyTime_GetMonotonicClock(); + return deadline - now; +} diff --git a/Python/thread_nt.h b/Python/thread_nt.h index 26f054ff0ce49..0dde1a0409738 100644 --- a/Python/thread_nt.h +++ b/Python/thread_nt.h @@ -75,20 +75,20 @@ EnterNonRecursiveMutex(PNRMUTEX mutex, DWORD milliseconds) } } } else if (milliseconds != 0) { - /* wait at least until the target */ - _PyTime_t now = _PyTime_GetPerfCounter(); + /* wait at least until the deadline */ _PyTime_t nanoseconds = _PyTime_FromNanoseconds((_PyTime_t)milliseconds * 1000000); - _PyTime_t target = now + nanoseconds; + _PyTime_t deadline = _PyTime_Add(_PyTime_GetPerfCounter(), nanoseconds); while (mutex->locked) { - _PyTime_t microseconds = _PyTime_AsMicroseconds(nanoseconds, _PyTime_ROUND_TIMEOUT); + _PyTime_t microseconds = _PyTime_AsMicroseconds(nanoseconds, + _PyTime_ROUND_TIMEOUT); if (PyCOND_TIMEDWAIT(&mutex->cv, &mutex->cs, microseconds) < 0) { result = WAIT_FAILED; break; } - now = _PyTime_GetPerfCounter(); - if (target <= now) + nanoseconds = deadline - _PyTime_GetPerfCounter(); + if (nanoseconds <= 0) { break; - nanoseconds = target - now; + } } } if (!mutex->locked) { diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index 9b5e273f1a8ba..12dad7e9e4427 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -438,7 +438,7 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, dprintf(("PyThread_acquire_lock_timed(%p, %lld, %d) called\n", lock, microseconds, intr_flag)); - _PyTime_t timeout; + _PyTime_t timeout; // relative timeout if (microseconds >= 0) { _PyTime_t ns; if (microseconds <= _PyTime_MAX / 1000) { @@ -465,16 +465,13 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, struct timespec abs_timeout; // Local scope for deadline { - _PyTime_t deadline = _PyTime_GetMonotonicClock() + timeout; + _PyTime_t deadline = _PyTime_Add(_PyTime_GetMonotonicClock(), timeout); _PyTime_AsTimespec_clamp(deadline, &abs_timeout); } #else _PyTime_t deadline = 0; - if (timeout > 0 - && !intr_flag - ) - { - deadline = _PyTime_GetMonotonicClock() + timeout; + if (timeout > 0 && !intr_flag) { + deadline = _PyDeadline_Init(timeout); } #endif @@ -484,9 +481,10 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, status = fix_status(sem_clockwait(thelock, CLOCK_MONOTONIC, &abs_timeout)); #else - _PyTime_t abs_timeout = _PyTime_GetSystemClock() + timeout; + _PyTime_t abs_time = _PyTime_Add(_PyTime_GetSystemClock(), + timeout); struct timespec ts; - _PyTime_AsTimespec_clamp(abs_timeout, &ts); + _PyTime_AsTimespec_clamp(abs_time, &ts); status = fix_status(sem_timedwait(thelock, &ts)); #endif } @@ -508,7 +506,7 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, #ifndef HAVE_SEM_CLOCKWAIT if (timeout > 0) { /* wait interrupted by a signal (EINTR): recompute the timeout */ - _PyTime_t timeout = deadline - _PyTime_GetMonotonicClock(); + _PyTime_t timeout = _PyDeadline_Get(deadline); if (timeout < 0) { status = ETIMEDOUT; break; From webhook-mailer at python.org Fri Oct 1 08:38:05 2021 From: webhook-mailer at python.org (JulienPalard) Date: Fri, 01 Oct 2021 12:38:05 -0000 Subject: [Python-checkins] sqlite3: Modernize documentation around unicode and bytes. (GH-28652) Message-ID: https://github.com/python/cpython/commit/1dac95c814763eb8a53896ac4326d8d51895d43d commit: 1dac95c814763eb8a53896ac4326d8d51895d43d branch: main author: Julien Palard committer: JulienPalard date: 2021-10-01T14:37:56+02:00 summary: sqlite3: Modernize documentation around unicode and bytes. (GH-28652) files: M Doc/includes/sqlite3/text_factory.py M Doc/library/sqlite3.rst diff --git a/Doc/includes/sqlite3/text_factory.py b/Doc/includes/sqlite3/text_factory.py index a857a155cdd4f..c0d87cd559118 100644 --- a/Doc/includes/sqlite3/text_factory.py +++ b/Doc/includes/sqlite3/text_factory.py @@ -3,9 +3,9 @@ con = sqlite3.connect(":memory:") cur = con.cursor() -AUSTRIA = "\xd6sterreich" +AUSTRIA = "?sterreich" -# by default, rows are returned as Unicode +# by default, rows are returned as str cur.execute("select ?", (AUSTRIA,)) row = cur.fetchone() assert row[0] == AUSTRIA diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index e6b8b95d2aa52..eaea7ae390b97 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -537,8 +537,8 @@ Connection Objects Using this attribute you can control what objects are returned for the ``TEXT`` data type. By default, this attribute is set to :class:`str` and the - :mod:`sqlite3` module will return Unicode objects for ``TEXT``. If you want to - return bytestrings instead, you can set it to :class:`bytes`. + :mod:`sqlite3` module will return :class:`str` objects for ``TEXT``. + If you want to return :class:`bytes` instead, you can set it to :class:`bytes`. You can also set it to any other callable that accepts a single bytestring parameter and returns the resulting object. From webhook-mailer at python.org Fri Oct 1 08:38:54 2021 From: webhook-mailer at python.org (JulienPalard) Date: Fri, 01 Oct 2021 12:38:54 -0000 Subject: [Python-checkins] hashlib: Fix old message about unicode objects. (GH-28653) Message-ID: https://github.com/python/cpython/commit/9ce0f48e918860ffa32751a85b0fe7967723e2e3 commit: 9ce0f48e918860ffa32751a85b0fe7967723e2e3 branch: main author: Julien Palard committer: JulienPalard date: 2021-10-01T14:38:49+02:00 summary: hashlib: Fix old message about unicode objects. (GH-28653) files: M Modules/hashlib.h diff --git a/Modules/hashlib.h b/Modules/hashlib.h index 978593e2f1a0c..56ae7a5e50bf5 100644 --- a/Modules/hashlib.h +++ b/Modules/hashlib.h @@ -8,7 +8,7 @@ #define GET_BUFFER_VIEW_OR_ERROR(obj, viewp, erraction) do { \ if (PyUnicode_Check((obj))) { \ PyErr_SetString(PyExc_TypeError, \ - "Unicode-objects must be encoded before hashing");\ + "Strings must be encoded before hashing");\ erraction; \ } \ if (!PyObject_CheckBuffer((obj))) { \ From webhook-mailer at python.org Fri Oct 1 08:58:46 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 12:58:46 -0000 Subject: [Python-checkins] hashlib: Fix old message about unicode objects. (GH-28653) Message-ID: https://github.com/python/cpython/commit/b994feeca78f8ff81666b0e8a40fe2003986f6c7 commit: b994feeca78f8ff81666b0e8a40fe2003986f6c7 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T05:58:37-07:00 summary: hashlib: Fix old message about unicode objects. (GH-28653) (cherry picked from commit 9ce0f48e918860ffa32751a85b0fe7967723e2e3) Co-authored-by: Julien Palard files: M Modules/hashlib.h diff --git a/Modules/hashlib.h b/Modules/hashlib.h index 978593e2f1a0c..56ae7a5e50bf5 100644 --- a/Modules/hashlib.h +++ b/Modules/hashlib.h @@ -8,7 +8,7 @@ #define GET_BUFFER_VIEW_OR_ERROR(obj, viewp, erraction) do { \ if (PyUnicode_Check((obj))) { \ PyErr_SetString(PyExc_TypeError, \ - "Unicode-objects must be encoded before hashing");\ + "Strings must be encoded before hashing");\ erraction; \ } \ if (!PyObject_CheckBuffer((obj))) { \ From webhook-mailer at python.org Fri Oct 1 09:09:24 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 13:09:24 -0000 Subject: [Python-checkins] bpo-45125: Improves pickling docs and tests for `shared_memory` (GH-28294) Message-ID: https://github.com/python/cpython/commit/fc3511f18e92ea345092aa59a4225218bd19e153 commit: fc3511f18e92ea345092aa59a4225218bd19e153 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T06:09:15-07:00 summary: bpo-45125: Improves pickling docs and tests for `shared_memory` (GH-28294) (cherry picked from commit 746d648d47d12d16c2afedaeff626fc6aaaf6a46) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst M Doc/library/multiprocessing.shared_memory.rst M Lib/test/_test_multiprocessing.py diff --git a/Doc/library/multiprocessing.shared_memory.rst b/Doc/library/multiprocessing.shared_memory.rst index cba576a29e2e2..2ba42b7e579a7 100644 --- a/Doc/library/multiprocessing.shared_memory.rst +++ b/Doc/library/multiprocessing.shared_memory.rst @@ -342,3 +342,30 @@ behind it: >>> c.shm.close() >>> c.shm.unlink() +The following examples demonstrates that ``ShareableList`` +(and underlying ``SharedMemory``) objects +can be pickled and unpickled if needed. +Note, that it will still be the same shared object. +This happens, because the deserialized object has +the same unique name and is just attached to an existing +object with the same name (if the object is still alive): + + >>> import pickle + >>> from multiprocessing import shared_memory + >>> sl = shared_memory.ShareableList(range(10)) + >>> list(sl) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> deserialized_sl = pickle.loads(pickle.dumps(sl)) + >>> list(deserialized_sl) + [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> sl[0] = -1 + >>> deserialized_sl[1] = -2 + >>> list(sl) + [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] + >>> list(deserialized_sl) + [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] + + >>> sl.shm.close() + >>> sl.shm.unlink() diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index 9e0d18da50f6e..3bc5b8f3d79b0 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3793,13 +3793,6 @@ def test_shared_memory_basics(self): self.assertIn(sms.name, str(sms)) self.assertIn(str(sms.size), str(sms)) - # Test pickling - sms.buf[0:6] = b'pickle' - pickled_sms = pickle.dumps(sms) - sms2 = pickle.loads(pickled_sms) - self.assertEqual(sms.name, sms2.name) - self.assertEqual(bytes(sms.buf[0:6]), bytes(sms2.buf[0:6]), b'pickle') - # Modify contents of shared memory segment through memoryview. sms.buf[0] = 42 self.assertEqual(sms.buf[0], 42) @@ -3898,6 +3891,29 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): sms.close() + def test_shared_memory_recreate(self): + # Test if shared memory segment is created properly, + # when _make_filename returns an existing shared memory segment name + with unittest.mock.patch( + 'multiprocessing.shared_memory._make_filename') as mock_make_filename: + + NAME_PREFIX = shared_memory._SHM_NAME_PREFIX + names = ['test01_fn', 'test02_fn'] + # Prepend NAME_PREFIX which can be '/psm_' or 'wnsm_', necessary + # because some POSIX compliant systems require name to start with / + names = [NAME_PREFIX + name for name in names] + + mock_make_filename.side_effect = names + shm1 = shared_memory.SharedMemory(create=True, size=1) + self.addCleanup(shm1.unlink) + self.assertEqual(shm1._name, names[0]) + + mock_make_filename.side_effect = names + shm2 = shared_memory.SharedMemory(create=True, size=1) + self.addCleanup(shm2.unlink) + self.assertEqual(shm2._name, names[1]) + + def test_invalid_shared_memory_cration(self): # Test creating a shared memory segment with negative size with self.assertRaises(ValueError): sms_invalid = shared_memory.SharedMemory(create=True, size=-1) @@ -3910,6 +3926,47 @@ class OptionalAttachSharedMemory(shared_memory.SharedMemory): with self.assertRaises(ValueError): sms_invalid = shared_memory.SharedMemory(create=True) + def test_shared_memory_pickle_unpickle(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sms = shared_memory.SharedMemory(create=True, size=512) + self.addCleanup(sms.unlink) + sms.buf[0:6] = b'pickle' + + # Test pickling + pickled_sms = pickle.dumps(sms, protocol=proto) + + # Test unpickling + sms2 = pickle.loads(pickled_sms) + self.assertIsInstance(sms2, shared_memory.SharedMemory) + self.assertEqual(sms.name, sms2.name) + self.assertEqual(bytes(sms.buf[0:6]), b'pickle') + self.assertEqual(bytes(sms2.buf[0:6]), b'pickle') + + # Test that unpickled version is still the same SharedMemory + sms.buf[0:6] = b'newval' + self.assertEqual(bytes(sms.buf[0:6]), b'newval') + self.assertEqual(bytes(sms2.buf[0:6]), b'newval') + + sms2.buf[0:6] = b'oldval' + self.assertEqual(bytes(sms.buf[0:6]), b'oldval') + self.assertEqual(bytes(sms2.buf[0:6]), b'oldval') + + def test_shared_memory_pickle_unpickle_dead_object(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sms = shared_memory.SharedMemory(create=True, size=512) + sms.buf[0:6] = b'pickle' + pickled_sms = pickle.dumps(sms, protocol=proto) + + # Now, we are going to kill the original object. + # So, unpickled one won't be able to attach to it. + sms.close() + sms.unlink() + + with self.assertRaises(FileNotFoundError): + pickle.loads(pickled_sms) + def test_shared_memory_across_processes(self): # bpo-40135: don't define shared memory block's name in case of # the failure when we run multiprocessing tests in parallel. @@ -4127,29 +4184,45 @@ def test_shared_memory_ShareableList_basics(self): empty_sl.shm.unlink() def test_shared_memory_ShareableList_pickling(self): - sl = shared_memory.ShareableList(range(10)) - self.addCleanup(sl.shm.unlink) - - serialized_sl = pickle.dumps(sl) - deserialized_sl = pickle.loads(serialized_sl) - self.assertTrue( - isinstance(deserialized_sl, shared_memory.ShareableList) - ) - self.assertTrue(deserialized_sl[-1], 9) - self.assertFalse(sl is deserialized_sl) - deserialized_sl[4] = "changed" - self.assertEqual(sl[4], "changed") - - # Verify data is not being put into the pickled representation. - name = 'a' * len(sl.shm.name) - larger_sl = shared_memory.ShareableList(range(400)) - self.addCleanup(larger_sl.shm.unlink) - serialized_larger_sl = pickle.dumps(larger_sl) - self.assertTrue(len(serialized_sl) == len(serialized_larger_sl)) - larger_sl.shm.close() - - deserialized_sl.shm.close() - sl.shm.close() + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sl = shared_memory.ShareableList(range(10)) + self.addCleanup(sl.shm.unlink) + + serialized_sl = pickle.dumps(sl, protocol=proto) + deserialized_sl = pickle.loads(serialized_sl) + self.assertIsInstance( + deserialized_sl, shared_memory.ShareableList) + self.assertEqual(deserialized_sl[-1], 9) + self.assertIsNot(sl, deserialized_sl) + + deserialized_sl[4] = "changed" + self.assertEqual(sl[4], "changed") + sl[3] = "newvalue" + self.assertEqual(deserialized_sl[3], "newvalue") + + larger_sl = shared_memory.ShareableList(range(400)) + self.addCleanup(larger_sl.shm.unlink) + serialized_larger_sl = pickle.dumps(larger_sl, protocol=proto) + self.assertEqual(len(serialized_sl), len(serialized_larger_sl)) + larger_sl.shm.close() + + deserialized_sl.shm.close() + sl.shm.close() + + def test_shared_memory_ShareableList_pickling_dead_object(self): + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + sl = shared_memory.ShareableList(range(10)) + serialized_sl = pickle.dumps(sl, protocol=proto) + + # Now, we are going to kill the original object. + # So, unpickled one won't be able to attach to it. + sl.shm.close() + sl.shm.unlink() + + with self.assertRaises(FileNotFoundError): + pickle.loads(serialized_sl) def test_shared_memory_cleaned_after_process_termination(self): cmd = '''if 1: @@ -4202,7 +4275,7 @@ def test_shared_memory_cleaned_after_process_termination(self): "shared_memory objects to clean up at shutdown", err) # -# +# Test to verify that `Finalize` works. # class _TestFinalize(BaseTestCase): diff --git a/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst b/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst new file mode 100644 index 0000000000000..5dfbe0e5db463 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst @@ -0,0 +1,2 @@ +Improves pickling tests and docs of ``SharedMemory`` and ``SharableList`` +objects. From webhook-mailer at python.org Fri Oct 1 09:28:28 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 13:28:28 -0000 Subject: [Python-checkins] [3.9] hashlib: Fix old message about unicode objects. (GH-28653) (GH-28679) Message-ID: https://github.com/python/cpython/commit/8822526caac80a0ab5f0b4722fe947e78c2ada7c commit: 8822526caac80a0ab5f0b4722fe947e78c2ada7c branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T06:28:19-07:00 summary: [3.9] hashlib: Fix old message about unicode objects. (GH-28653) (GH-28679) (cherry picked from commit 9ce0f48e918860ffa32751a85b0fe7967723e2e3) Co-authored-by: Julien Palard files: M Modules/hashlib.h diff --git a/Modules/hashlib.h b/Modules/hashlib.h index 978593e2f1a0c..56ae7a5e50bf5 100644 --- a/Modules/hashlib.h +++ b/Modules/hashlib.h @@ -8,7 +8,7 @@ #define GET_BUFFER_VIEW_OR_ERROR(obj, viewp, erraction) do { \ if (PyUnicode_Check((obj))) { \ PyErr_SetString(PyExc_TypeError, \ - "Unicode-objects must be encoded before hashing");\ + "Strings must be encoded before hashing");\ erraction; \ } \ if (!PyObject_CheckBuffer((obj))) { \ From webhook-mailer at python.org Fri Oct 1 09:49:55 2021 From: webhook-mailer at python.org (corona10) Date: Fri, 01 Oct 2021 13:49:55 -0000 Subject: [Python-checkins] bpo-45332: Fix broken Decimal test and benchmark (GH-28680) Message-ID: https://github.com/python/cpython/commit/9eed75fde226cec5a02301cfac1dc8039b5a183e commit: 9eed75fde226cec5a02301cfac1dc8039b5a183e branch: main author: Dong-hee Na committer: corona10 date: 2021-10-01T22:49:46+09:00 summary: bpo-45332: Fix broken Decimal test and benchmark (GH-28680) files: M Modules/_decimal/tests/bench.py M Modules/_decimal/tests/deccheck.py M Modules/_decimal/tests/formathelper.py diff --git a/Modules/_decimal/tests/bench.py b/Modules/_decimal/tests/bench.py index 3726db194e032..24e091b6887cc 100644 --- a/Modules/_decimal/tests/bench.py +++ b/Modules/_decimal/tests/bench.py @@ -7,10 +7,7 @@ import time -try: - from test.support import import_fresh_module -except ImportError: - from test.test_support import import_fresh_module +from test.support.import_helper import import_fresh_module C = import_fresh_module('decimal', fresh=['_decimal']) P = import_fresh_module('decimal', blocked=['_decimal']) diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index 98ecd502c03fe..edf753f3704a1 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -47,7 +47,7 @@ from queue import Queue, Empty from threading import Thread, Event, Lock -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module from randdec import randfloat, all_unary, all_binary, all_ternary from randdec import unary_optarg, binary_optarg, ternary_optarg from formathelper import rand_format, rand_locale diff --git a/Modules/_decimal/tests/formathelper.py b/Modules/_decimal/tests/formathelper.py index 19b2aad4a503b..c3daacfb7b44f 100644 --- a/Modules/_decimal/tests/formathelper.py +++ b/Modules/_decimal/tests/formathelper.py @@ -31,7 +31,7 @@ import os, sys, locale, random import platform, subprocess -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module from distutils.spawn import find_executable C = import_fresh_module('decimal', fresh=['_decimal']) From webhook-mailer at python.org Fri Oct 1 10:16:45 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 14:16:45 -0000 Subject: [Python-checkins] bpo-45332: Fix broken Decimal test and benchmark (GH-28680) Message-ID: https://github.com/python/cpython/commit/282992b36f9fe27183037051f3b37210884600af commit: 282992b36f9fe27183037051f3b37210884600af branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T07:16:37-07:00 summary: bpo-45332: Fix broken Decimal test and benchmark (GH-28680) (cherry picked from commit 9eed75fde226cec5a02301cfac1dc8039b5a183e) Co-authored-by: Dong-hee Na files: M Modules/_decimal/tests/bench.py M Modules/_decimal/tests/deccheck.py M Modules/_decimal/tests/formathelper.py diff --git a/Modules/_decimal/tests/bench.py b/Modules/_decimal/tests/bench.py index 3726db194e032..24e091b6887cc 100644 --- a/Modules/_decimal/tests/bench.py +++ b/Modules/_decimal/tests/bench.py @@ -7,10 +7,7 @@ import time -try: - from test.support import import_fresh_module -except ImportError: - from test.test_support import import_fresh_module +from test.support.import_helper import import_fresh_module C = import_fresh_module('decimal', fresh=['_decimal']) P = import_fresh_module('decimal', blocked=['_decimal']) diff --git a/Modules/_decimal/tests/deccheck.py b/Modules/_decimal/tests/deccheck.py index 98ecd502c03fe..edf753f3704a1 100644 --- a/Modules/_decimal/tests/deccheck.py +++ b/Modules/_decimal/tests/deccheck.py @@ -47,7 +47,7 @@ from queue import Queue, Empty from threading import Thread, Event, Lock -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module from randdec import randfloat, all_unary, all_binary, all_ternary from randdec import unary_optarg, binary_optarg, ternary_optarg from formathelper import rand_format, rand_locale diff --git a/Modules/_decimal/tests/formathelper.py b/Modules/_decimal/tests/formathelper.py index 19b2aad4a503b..c3daacfb7b44f 100644 --- a/Modules/_decimal/tests/formathelper.py +++ b/Modules/_decimal/tests/formathelper.py @@ -31,7 +31,7 @@ import os, sys, locale, random import platform, subprocess -from test.support import import_fresh_module +from test.support.import_helper import import_fresh_module from distutils.spawn import find_executable C = import_fresh_module('decimal', fresh=['_decimal']) From webhook-mailer at python.org Fri Oct 1 10:44:27 2021 From: webhook-mailer at python.org (markshannon) Date: Fri, 01 Oct 2021 14:44:27 -0000 Subject: [Python-checkins] Fix a couple of compiler warnings. (GH-28677) Message-ID: https://github.com/python/cpython/commit/cd760ceb67c5164983838ed7eb73a833d1597da0 commit: cd760ceb67c5164983838ed7eb73a833d1597da0 branch: main author: Mark Shannon committer: markshannon date: 2021-10-01T15:44:19+01:00 summary: Fix a couple of compiler warnings. (GH-28677) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index ab692fd8ded15..7f29967eb3272 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3666,7 +3666,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(PyDict_CheckExact((PyObject *)dict)); PyObject *name = GETITEM(names, cache0->original_oparg); uint32_t hint = cache1->dk_version_or_hint; - DEOPT_IF(hint >= dict->ma_keys->dk_nentries, LOAD_ATTR); + DEOPT_IF(hint >= (size_t)dict->ma_keys->dk_nentries, LOAD_ATTR); PyDictKeyEntry *ep = DK_ENTRIES(dict->ma_keys) + hint; DEOPT_IF(ep->me_key != name, LOAD_ATTR); res = ep->me_value; @@ -3774,7 +3774,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(PyDict_CheckExact((PyObject *)dict)); PyObject *name = GETITEM(names, cache0->original_oparg); uint32_t hint = cache1->dk_version_or_hint; - DEOPT_IF(hint >= dict->ma_keys->dk_nentries, STORE_ATTR); + DEOPT_IF(hint >= (size_t)dict->ma_keys->dk_nentries, STORE_ATTR); PyDictKeyEntry *ep = DK_ENTRIES(dict->ma_keys) + hint; DEOPT_IF(ep->me_key != name, STORE_ATTR); PyObject *old_value = ep->me_value; From webhook-mailer at python.org Fri Oct 1 12:22:58 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 16:22:58 -0000 Subject: [Python-checkins] bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28671) Message-ID: https://github.com/python/cpython/commit/6df8c327532627d6a99991993c52e8e4a9b34968 commit: 6df8c327532627d6a99991993c52e8e4a9b34968 branch: 3.10 author: Victor Stinner committer: vstinner date: 2021-10-01T18:22:49+02:00 summary: bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28671) On Unix, if the sem_clockwait() function is available in the C library (glibc 2.30 and newer), the threading.Lock.acquire() method now uses the monotonic clock (time.CLOCK_MONOTONIC) for the timeout, rather than using the system clock (time.CLOCK_REALTIME), to not be affected by system clock changes. configure now checks if the sem_clockwait() function is available. files: A Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst M Python/thread_pthread.h M configure M configure.ac M pyconfig.h.in diff --git a/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst new file mode 100644 index 0000000000000..d8a4f9507c189 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst @@ -0,0 +1,5 @@ +On Unix, if the ``sem_clockwait()`` function is available in the C library +(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses the +monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather than +using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected by +system clock changes. Patch by Victor Stinner. diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index a45d842ffe73d..35b9810aa377f 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -92,12 +92,17 @@ * mutexes and condition variables: */ #if (defined(_POSIX_SEMAPHORES) && !defined(HAVE_BROKEN_POSIX_SEMAPHORES) && \ - defined(HAVE_SEM_TIMEDWAIT)) + (defined(HAVE_SEM_TIMEDWAIT) || defined(HAVE_SEM_CLOCKWAIT))) # define USE_SEMAPHORES #else # undef USE_SEMAPHORES #endif +#if defined(HAVE_PTHREAD_CONDATTR_SETCLOCK) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) +// monotonic is supported statically. It doesn't mean it works on runtime. +#define CONDATTR_MONOTONIC +#endif + /* On platforms that don't use standard POSIX threads pthread_sigmask() * isn't present. DEC threads uses sigprocmask() instead as do most @@ -123,16 +128,23 @@ do { \ ts.tv_nsec = tv.tv_usec * 1000; \ } while(0) +#if defined(CONDATTR_MONOTONIC) || defined(HAVE_SEM_CLOCKWAIT) +static void +monotonic_abs_timeout(long long us, struct timespec *abs) +{ + clock_gettime(CLOCK_MONOTONIC, abs); + abs->tv_sec += us / 1000000; + abs->tv_nsec += (us % 1000000) * 1000; + abs->tv_sec += abs->tv_nsec / 1000000000; + abs->tv_nsec %= 1000000000; +} +#endif + /* * pthread_cond support */ -#if defined(HAVE_PTHREAD_CONDATTR_SETCLOCK) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) -// monotonic is supported statically. It doesn't mean it works on runtime. -#define CONDATTR_MONOTONIC -#endif - // NULL when pthread_condattr_setclock(CLOCK_MONOTONIC) is not supported. static pthread_condattr_t *condattr_monotonic = NULL; @@ -154,16 +166,13 @@ _PyThread_cond_init(PyCOND_T *cond) return pthread_cond_init(cond, condattr_monotonic); } + void _PyThread_cond_after(long long us, struct timespec *abs) { #ifdef CONDATTR_MONOTONIC if (condattr_monotonic) { - clock_gettime(CLOCK_MONOTONIC, abs); - abs->tv_sec += us / 1000000; - abs->tv_nsec += (us % 1000000) * 1000; - abs->tv_sec += abs->tv_nsec / 1000000000; - abs->tv_nsec %= 1000000000; + monotonic_abs_timeout(us, abs); return; } #endif @@ -434,7 +443,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, sem_t *thelock = (sem_t *)lock; int status, error = 0; struct timespec ts; +#ifndef HAVE_SEM_CLOCKWAIT _PyTime_t deadline = 0; +#endif (void) error; /* silence unused-but-set-variable warning */ dprintf(("PyThread_acquire_lock_timed(%p, %lld, %d) called\n", @@ -445,6 +456,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, } if (microseconds > 0) { +#ifdef HAVE_SEM_CLOCKWAIT + monotonic_abs_timeout(microseconds, &ts); +#else MICROSECONDS_TO_TIMESPEC(microseconds, ts); if (!intr_flag) { @@ -453,11 +467,17 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, _PyTime_t timeout = _PyTime_FromNanoseconds(microseconds * 1000); deadline = _PyTime_GetMonotonicClock() + timeout; } +#endif } while (1) { if (microseconds > 0) { +#ifdef HAVE_SEM_CLOCKWAIT + status = fix_status(sem_clockwait(thelock, CLOCK_MONOTONIC, + &ts)); +#else status = fix_status(sem_timedwait(thelock, &ts)); +#endif } else if (microseconds == 0) { status = fix_status(sem_trywait(thelock)); @@ -472,6 +492,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, break; } + // sem_clockwait() uses an absolute timeout, there is no need + // to recompute the relative timeout. +#ifndef HAVE_SEM_CLOCKWAIT if (microseconds > 0) { /* wait interrupted by a signal (EINTR): recompute the timeout */ _PyTime_t dt = deadline - _PyTime_GetMonotonicClock(); @@ -493,13 +516,19 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, microseconds = 0; } } +#endif } /* Don't check the status if we're stopping because of an interrupt. */ if (!(intr_flag && status == EINTR)) { if (microseconds > 0) { - if (status != ETIMEDOUT) + if (status != ETIMEDOUT) { +#ifdef HAVE_SEM_CLOCKWAIT + CHECK_STATUS("sem_clockwait"); +#else CHECK_STATUS("sem_timedwait"); +#endif + } } else if (microseconds == 0) { if (status != EAGAIN) diff --git a/configure b/configure index 15801175a3dd5..753f9564693bc 100755 --- a/configure +++ b/configure @@ -11738,7 +11738,7 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ diff --git a/configure.ac b/configure.ac index a75fdf918c7ea..41a3679d3dc55 100644 --- a/configure.ac +++ b/configure.ac @@ -3707,7 +3707,7 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ diff --git a/pyconfig.h.in b/pyconfig.h.in index 49407ab62b417..0559274b4768c 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -902,6 +902,9 @@ /* Define to 1 if you have the `sched_setscheduler' function. */ #undef HAVE_SCHED_SETSCHEDULER +/* Define to 1 if you have the `sem_clockwait' function. */ +#undef HAVE_SEM_CLOCKWAIT + /* Define to 1 if you have the `sem_getvalue' function. */ #undef HAVE_SEM_GETVALUE From webhook-mailer at python.org Fri Oct 1 12:51:28 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 01 Oct 2021 16:51:28 -0000 Subject: [Python-checkins] bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28671) (GH-28683) Message-ID: https://github.com/python/cpython/commit/0e1aeab5d7de3f328876aea8ccabbc6db146a883 commit: 0e1aeab5d7de3f328876aea8ccabbc6db146a883 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vstinner date: 2021-10-01T18:51:15+02:00 summary: bpo-41710: PyThread_acquire_lock_timed() uses sem_clockwait() (GH-28671) (GH-28683) On Unix, if the sem_clockwait() function is available in the C library (glibc 2.30 and newer), the threading.Lock.acquire() method now uses the monotonic clock (time.CLOCK_MONOTONIC) for the timeout, rather than using the system clock (time.CLOCK_REALTIME), to not be affected by system clock changes. configure now checks if the sem_clockwait() function is available. (cherry picked from commit 6df8c327532627d6a99991993c52e8e4a9b34968) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst M Python/thread_pthread.h M configure M configure.ac M pyconfig.h.in diff --git a/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst new file mode 100644 index 0000000000000..d8a4f9507c189 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst @@ -0,0 +1,5 @@ +On Unix, if the ``sem_clockwait()`` function is available in the C library +(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses the +monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather than +using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected by +system clock changes. Patch by Victor Stinner. diff --git a/Python/thread_pthread.h b/Python/thread_pthread.h index e6910b3083a89..83974b4c0ca3b 100644 --- a/Python/thread_pthread.h +++ b/Python/thread_pthread.h @@ -89,12 +89,17 @@ * mutexes and condition variables: */ #if (defined(_POSIX_SEMAPHORES) && !defined(HAVE_BROKEN_POSIX_SEMAPHORES) && \ - defined(HAVE_SEM_TIMEDWAIT)) + (defined(HAVE_SEM_TIMEDWAIT) || defined(HAVE_SEM_CLOCKWAIT))) # define USE_SEMAPHORES #else # undef USE_SEMAPHORES #endif +#if defined(HAVE_PTHREAD_CONDATTR_SETCLOCK) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) +// monotonic is supported statically. It doesn't mean it works on runtime. +#define CONDATTR_MONOTONIC +#endif + /* On platforms that don't use standard POSIX threads pthread_sigmask() * isn't present. DEC threads uses sigprocmask() instead as do most @@ -120,16 +125,23 @@ do { \ ts.tv_nsec = tv.tv_usec * 1000; \ } while(0) +#if defined(CONDATTR_MONOTONIC) || defined(HAVE_SEM_CLOCKWAIT) +static void +monotonic_abs_timeout(long long us, struct timespec *abs) +{ + clock_gettime(CLOCK_MONOTONIC, abs); + abs->tv_sec += us / 1000000; + abs->tv_nsec += (us % 1000000) * 1000; + abs->tv_sec += abs->tv_nsec / 1000000000; + abs->tv_nsec %= 1000000000; +} +#endif + /* * pthread_cond support */ -#if defined(HAVE_PTHREAD_CONDATTR_SETCLOCK) && defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) -// monotonic is supported statically. It doesn't mean it works on runtime. -#define CONDATTR_MONOTONIC -#endif - // NULL when pthread_condattr_setclock(CLOCK_MONOTONIC) is not supported. static pthread_condattr_t *condattr_monotonic = NULL; @@ -151,16 +163,13 @@ _PyThread_cond_init(PyCOND_T *cond) return pthread_cond_init(cond, condattr_monotonic); } + void _PyThread_cond_after(long long us, struct timespec *abs) { #ifdef CONDATTR_MONOTONIC if (condattr_monotonic) { - clock_gettime(CLOCK_MONOTONIC, abs); - abs->tv_sec += us / 1000000; - abs->tv_nsec += (us % 1000000) * 1000; - abs->tv_sec += abs->tv_nsec / 1000000000; - abs->tv_nsec %= 1000000000; + monotonic_abs_timeout(us, abs); return; } #endif @@ -431,7 +440,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, sem_t *thelock = (sem_t *)lock; int status, error = 0; struct timespec ts; +#ifndef HAVE_SEM_CLOCKWAIT _PyTime_t deadline = 0; +#endif (void) error; /* silence unused-but-set-variable warning */ dprintf(("PyThread_acquire_lock_timed(%p, %lld, %d) called\n", @@ -442,6 +453,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, } if (microseconds > 0) { +#ifdef HAVE_SEM_CLOCKWAIT + monotonic_abs_timeout(microseconds, &ts); +#else MICROSECONDS_TO_TIMESPEC(microseconds, ts); if (!intr_flag) { @@ -450,11 +464,17 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, _PyTime_t timeout = _PyTime_FromNanoseconds(microseconds * 1000); deadline = _PyTime_GetMonotonicClock() + timeout; } +#endif } while (1) { if (microseconds > 0) { +#ifdef HAVE_SEM_CLOCKWAIT + status = fix_status(sem_clockwait(thelock, CLOCK_MONOTONIC, + &ts)); +#else status = fix_status(sem_timedwait(thelock, &ts)); +#endif } else if (microseconds == 0) { status = fix_status(sem_trywait(thelock)); @@ -469,6 +489,9 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, break; } + // sem_clockwait() uses an absolute timeout, there is no need + // to recompute the relative timeout. +#ifndef HAVE_SEM_CLOCKWAIT if (microseconds > 0) { /* wait interrupted by a signal (EINTR): recompute the timeout */ _PyTime_t dt = deadline - _PyTime_GetMonotonicClock(); @@ -490,13 +513,19 @@ PyThread_acquire_lock_timed(PyThread_type_lock lock, PY_TIMEOUT_T microseconds, microseconds = 0; } } +#endif } /* Don't check the status if we're stopping because of an interrupt. */ if (!(intr_flag && status == EINTR)) { if (microseconds > 0) { - if (status != ETIMEDOUT) + if (status != ETIMEDOUT) { +#ifdef HAVE_SEM_CLOCKWAIT + CHECK_STATUS("sem_clockwait"); +#else CHECK_STATUS("sem_timedwait"); +#endif + } } else if (microseconds == 0) { if (status != EAGAIN) diff --git a/configure b/configure index ffa61c1dc5126..7cad0e2f98ba1 100755 --- a/configure +++ b/configure @@ -11723,7 +11723,7 @@ for ac_func in alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ diff --git a/configure.ac b/configure.ac index 8fe5fa5742e8b..be28e3a38f7ab 100644 --- a/configure.ac +++ b/configure.ac @@ -3707,7 +3707,7 @@ AC_CHECK_FUNCS(alarm accept4 setitimer getitimer bind_textdomain_codeset chown \ posix_fallocate posix_fadvise posix_spawn posix_spawnp pread preadv preadv2 \ pthread_condattr_setclock pthread_init pthread_kill pwrite pwritev pwritev2 \ readlink readlinkat readv realpath renameat \ - sem_open sem_timedwait sem_getvalue sem_unlink sendfile setegid seteuid \ + sem_open sem_timedwait sem_clockwait sem_getvalue sem_unlink sendfile setegid seteuid \ setgid sethostname \ setlocale setregid setreuid setresuid setresgid setsid setpgid setpgrp setpriority setuid setvbuf \ sched_get_priority_max sched_setaffinity sched_setscheduler sched_setparam \ diff --git a/pyconfig.h.in b/pyconfig.h.in index 6358e568f4a6f..b62e169255d99 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -887,6 +887,9 @@ /* Define to 1 if you have the `sched_setscheduler' function. */ #undef HAVE_SCHED_SETSCHEDULER +/* Define to 1 if you have the `sem_clockwait' function. */ +#undef HAVE_SEM_CLOCKWAIT + /* Define to 1 if you have the `sem_getvalue' function. */ #undef HAVE_SEM_GETVALUE From webhook-mailer at python.org Fri Oct 1 16:11:12 2021 From: webhook-mailer at python.org (zooba) Date: Fri, 01 Oct 2021 20:11:12 -0000 Subject: [Python-checkins] bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) Message-ID: https://github.com/python/cpython/commit/a450398933d265011e1e8eae7f771b70f97945fb commit: a450398933d265011e1e8eae7f771b70f97945fb branch: main author: AngstyDuck committer: zooba date: 2021-10-01T21:11:08+01:00 summary: bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) files: A Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst M Modules/_io/bufferedio.c diff --git a/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst new file mode 100644 index 0000000000000..d38fa6057f6f9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst @@ -0,0 +1 @@ +:meth:`BufferedReader.peek` no longer raises :exc:`ValueError` when the entire file has already been buffered. diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 5984d34cc0829..ba966f568b399 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -341,11 +341,10 @@ _enter_buffered_busy(buffered *self) : buffered_closed(self))) #define CHECK_CLOSED(self, error_msg) \ - if (IS_CLOSED(self)) { \ + if (IS_CLOSED(self) & (Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t) == 0)) { \ PyErr_SetString(PyExc_ValueError, error_msg); \ return NULL; \ - } - + } \ #define VALID_READ_BUFFER(self) \ (self->readable && self->read_end != -1) @@ -530,6 +529,9 @@ buffered_close(buffered *self, PyObject *args) Py_CLEAR(res); } + self->read_end = 0; + self->pos = 0; + end: LEAVE_BUFFERED(self) return res; From webhook-mailer at python.org Fri Oct 1 16:30:24 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 01 Oct 2021 20:30:24 -0000 Subject: [Python-checkins] bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) Message-ID: https://github.com/python/cpython/commit/2221207f44c2abd34406ee590e6b13afaca53ee8 commit: 2221207f44c2abd34406ee590e6b13afaca53ee8 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T13:30:15-07:00 summary: bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) (cherry picked from commit a450398933d265011e1e8eae7f771b70f97945fb) Co-authored-by: AngstyDuck files: A Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst M Modules/_io/bufferedio.c diff --git a/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst new file mode 100644 index 0000000000000..d38fa6057f6f9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst @@ -0,0 +1 @@ +:meth:`BufferedReader.peek` no longer raises :exc:`ValueError` when the entire file has already been buffered. diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 5984d34cc0829..ba966f568b399 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -341,11 +341,10 @@ _enter_buffered_busy(buffered *self) : buffered_closed(self))) #define CHECK_CLOSED(self, error_msg) \ - if (IS_CLOSED(self)) { \ + if (IS_CLOSED(self) & (Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t) == 0)) { \ PyErr_SetString(PyExc_ValueError, error_msg); \ return NULL; \ - } - + } \ #define VALID_READ_BUFFER(self) \ (self->readable && self->read_end != -1) @@ -530,6 +529,9 @@ buffered_close(buffered *self, PyObject *args) Py_CLEAR(res); } + self->read_end = 0; + self->pos = 0; + end: LEAVE_BUFFERED(self) return res; From webhook-mailer at python.org Fri Oct 1 16:46:32 2021 From: webhook-mailer at python.org (zooba) Date: Fri, 01 Oct 2021 20:46:32 -0000 Subject: [Python-checkins] bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) Message-ID: https://github.com/python/cpython/commit/6035d650a322cec9619b306af2a877f3cead1580 commit: 6035d650a322cec9619b306af2a877f3cead1580 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: zooba date: 2021-10-01T21:46:25+01:00 summary: bpo-44687: Ensure BufferedReader objects with unread buffers can peek even when the underlying file is closed (GH-28457) Co-authored-by: AngstyDuck files: A Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst M Modules/_io/bufferedio.c diff --git a/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst new file mode 100644 index 0000000000000..d38fa6057f6f9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst @@ -0,0 +1 @@ +:meth:`BufferedReader.peek` no longer raises :exc:`ValueError` when the entire file has already been buffered. diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index f8e21f206f316..b0fe9e4589102 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -341,11 +341,10 @@ _enter_buffered_busy(buffered *self) : buffered_closed(self))) #define CHECK_CLOSED(self, error_msg) \ - if (IS_CLOSED(self)) { \ + if (IS_CLOSED(self) & (Py_SAFE_DOWNCAST(READAHEAD(self), Py_off_t, Py_ssize_t) == 0)) { \ PyErr_SetString(PyExc_ValueError, error_msg); \ return NULL; \ - } - + } \ #define VALID_READ_BUFFER(self) \ (self->readable && self->read_end != -1) @@ -530,6 +529,9 @@ buffered_close(buffered *self, PyObject *args) Py_CLEAR(res); } + self->read_end = 0; + self->pos = 0; + end: LEAVE_BUFFERED(self) return res; From webhook-mailer at python.org Sat Oct 2 02:05:01 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 02 Oct 2021 06:05:01 -0000 Subject: [Python-checkins] bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (#28687) Message-ID: https://github.com/python/cpython/commit/0be338199fd663f020d833a4db185d0c5a0e0078 commit: 0be338199fd663f020d833a4db185d0c5a0e0078 branch: main author: Terry Jan Reedy committer: terryjreedy date: 2021-10-02T02:04:55-04:00 summary: bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (#28687) pypi.org " The Python Package Index (PyPI) ... files: M Doc/distributing/index.rst M Doc/installing/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst index 66ba1e9fb9b34..136cf4e77b154 100644 --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -31,7 +31,7 @@ installing other Python projects, refer to the Key terms ========= -* the `Python Packaging Index `__ is a public +* the `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority @@ -127,14 +127,14 @@ involved in creating and publishing a project: * `Project structure`_ * `Building and packaging the project`_ -* `Uploading the project to the Python Packaging Index`_ +* `Uploading the project to the Python Package Index`_ * `The .pypirc file`_ .. _Project structure: \ https://packaging.python.org/tutorials/packaging-projects/#packaging-python-projects .. _Building and packaging the project: \ https://packaging.python.org/tutorials/packaging-projects/#creating-the-package-files -.. _Uploading the project to the Python Packaging Index: \ +.. _Uploading the project to the Python Package Index: \ https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives .. _The .pypirc file: \ https://packaging.python.org/specifications/pypirc/ @@ -150,7 +150,7 @@ These are quick answers or links for some common tasks. This isn't an easy topic, but here are a few tips: -* check the Python Packaging Index to see if the name is already in use +* check the Python Package Index to see if the name is already in use * check popular hosting sites like GitHub, Bitbucket, etc to see if there is already a project with that name * check what comes up in a web search for the name you're considering diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 5e7e03045b2ac..4bacc7ba0c2cf 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -44,7 +44,7 @@ Key terms ``venv``. It allows virtual environments to be used on versions of Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. -* The `Python Packaging Index `__ is a public +* The `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users. * the `Python Packaging Authority @@ -78,7 +78,7 @@ The standard packaging tools are all designed to be used from the command line. The following command will install the latest version of a module and its -dependencies from the Python Packaging Index:: +dependencies from the Python Package Index:: python -m pip install SomePackage @@ -226,7 +226,7 @@ the installation process. With the introduction of support for the binary ``wheel`` format, and the ability to publish wheels for at least Windows and macOS through the -Python Packaging Index, this problem is expected to diminish over time, +Python Package Index, this problem is expected to diminish over time, as users are more regularly able to install pre-built extensions rather than needing to build them themselves. From webhook-mailer at python.org Sat Oct 2 02:27:31 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Oct 2021 06:27:31 -0000 Subject: [Python-checkins] bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (GH-28687) Message-ID: https://github.com/python/cpython/commit/e040adc806aba32c53f4c3d35899d0e5691cab95 commit: e040adc806aba32c53f4c3d35899d0e5691cab95 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-01T23:27:23-07:00 summary: bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (GH-28687) pypi.org " The Python Package Index (PyPI) ... (cherry picked from commit 0be338199fd663f020d833a4db185d0c5a0e0078) Co-authored-by: Terry Jan Reedy files: M Doc/distributing/index.rst M Doc/installing/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst index 66ba1e9fb9b34..136cf4e77b154 100644 --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -31,7 +31,7 @@ installing other Python projects, refer to the Key terms ========= -* the `Python Packaging Index `__ is a public +* the `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority @@ -127,14 +127,14 @@ involved in creating and publishing a project: * `Project structure`_ * `Building and packaging the project`_ -* `Uploading the project to the Python Packaging Index`_ +* `Uploading the project to the Python Package Index`_ * `The .pypirc file`_ .. _Project structure: \ https://packaging.python.org/tutorials/packaging-projects/#packaging-python-projects .. _Building and packaging the project: \ https://packaging.python.org/tutorials/packaging-projects/#creating-the-package-files -.. _Uploading the project to the Python Packaging Index: \ +.. _Uploading the project to the Python Package Index: \ https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives .. _The .pypirc file: \ https://packaging.python.org/specifications/pypirc/ @@ -150,7 +150,7 @@ These are quick answers or links for some common tasks. This isn't an easy topic, but here are a few tips: -* check the Python Packaging Index to see if the name is already in use +* check the Python Package Index to see if the name is already in use * check popular hosting sites like GitHub, Bitbucket, etc to see if there is already a project with that name * check what comes up in a web search for the name you're considering diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 5e7e03045b2ac..4bacc7ba0c2cf 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -44,7 +44,7 @@ Key terms ``venv``. It allows virtual environments to be used on versions of Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. -* The `Python Packaging Index `__ is a public +* The `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users. * the `Python Packaging Authority @@ -78,7 +78,7 @@ The standard packaging tools are all designed to be used from the command line. The following command will install the latest version of a module and its -dependencies from the Python Packaging Index:: +dependencies from the Python Package Index:: python -m pip install SomePackage @@ -226,7 +226,7 @@ the installation process. With the introduction of support for the binary ``wheel`` format, and the ability to publish wheels for at least Windows and macOS through the -Python Packaging Index, this problem is expected to diminish over time, +Python Package Index, this problem is expected to diminish over time, as users are more regularly able to install pre-built extensions rather than needing to build them themselves. From webhook-mailer at python.org Sat Oct 2 05:39:11 2021 From: webhook-mailer at python.org (JulienPalard) Date: Sat, 02 Oct 2021 09:39:11 -0000 Subject: [Python-checkins] Makefile: Fix missing slashes (GH-28659) Message-ID: https://github.com/python/cpython/commit/417faa69bd48dfc22e4eff9bb2c610f53d59d02f commit: 417faa69bd48dfc22e4eff9bb2c610f53d59d02f branch: main author: native-api committer: JulienPalard date: 2021-10-02T11:38:59+02:00 summary: Makefile: Fix missing slashes (GH-28659) files: A Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst M Makefile.pre.in diff --git a/Makefile.pre.in b/Makefile.pre.in index 670887437360c..ce75af1b79c81 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1443,13 +1443,13 @@ altbininstall: $(BUILDPYTHON) @FRAMEWORKPYTHONW@ fi; \ fi if test "x$(LIPO_32BIT_FLAGS)" != "x" ; then \ - rm -f $(DESTDIR)$(BINDIR)python$(VERSION)-32$(EXE); \ + rm -f $(DESTDIR)$(BINDIR)/python$(VERSION)-32$(EXE); \ lipo $(LIPO_32BIT_FLAGS) \ -output $(DESTDIR)$(BINDIR)/python$(VERSION)-32$(EXE) \ $(DESTDIR)$(BINDIR)/python$(VERSION)$(EXE); \ fi if test "x$(LIPO_INTEL64_FLAGS)" != "x" ; then \ - rm -f $(DESTDIR)$(BINDIR)python$(VERSION)-intel64$(EXE); \ + rm -f $(DESTDIR)$(BINDIR)/python$(VERSION)-intel64$(EXE); \ lipo $(LIPO_INTEL64_FLAGS) \ -output $(DESTDIR)$(BINDIR)/python$(VERSION)-intel64$(EXE) \ $(DESTDIR)$(BINDIR)/python$(VERSION)$(EXE); \ diff --git a/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst b/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst new file mode 100644 index 0000000000000..a0ab4baf7952b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst @@ -0,0 +1 @@ +Makefile: fix missing slashes in some invocations cleaning previous build results when builing a macOS universal binary. \ No newline at end of file From webhook-mailer at python.org Sat Oct 2 05:57:17 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 02 Oct 2021 09:57:17 -0000 Subject: [Python-checkins] bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) Message-ID: https://github.com/python/cpython/commit/0742abdc48886b74ed3b66985a54bb1c32802670 commit: 0742abdc48886b74ed3b66985a54bb1c32802670 branch: main author: TAGAMI Yukihiro committer: serhiy-storchaka date: 2021-10-02T12:57:13+03:00 summary: bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) files: A Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst M Modules/pyexpat.c diff --git a/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst b/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst new file mode 100644 index 0000000000000..b4bedbc278edf --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst @@ -0,0 +1,2 @@ +Fix freed memory access in :class:`pyexpat.xmlparser` when building it with an +installed expat library <= 2.2.0. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index ec684638ead11..b3d9bdda7e7ac 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1204,10 +1204,10 @@ static void xmlparse_dealloc(xmlparseobject *self) { PyObject_GC_UnTrack(self); + (void)xmlparse_clear(self); if (self->itself != NULL) XML_ParserFree(self->itself); self->itself = NULL; - (void)xmlparse_clear(self); if (self->handlers != NULL) { PyMem_Free(self->handlers); From webhook-mailer at python.org Sat Oct 2 09:21:59 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 02 Oct 2021 13:21:59 -0000 Subject: [Python-checkins] bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) Message-ID: https://github.com/python/cpython/commit/22cf6a2f2347b7d4f11e45e557beace55acc79b5 commit: 22cf6a2f2347b7d4f11e45e557beace55acc79b5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-02T06:21:55-07:00 summary: bpo-45329: Fix freed memory access in pyexpat.c (GH-28649) (cherry picked from commit 0742abdc48886b74ed3b66985a54bb1c32802670) Co-authored-by: TAGAMI Yukihiro files: A Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst M Modules/pyexpat.c diff --git a/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst b/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst new file mode 100644 index 0000000000000..b4bedbc278edf --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst @@ -0,0 +1,2 @@ +Fix freed memory access in :class:`pyexpat.xmlparser` when building it with an +installed expat library <= 2.2.0. diff --git a/Modules/pyexpat.c b/Modules/pyexpat.c index ec684638ead11..b3d9bdda7e7ac 100644 --- a/Modules/pyexpat.c +++ b/Modules/pyexpat.c @@ -1204,10 +1204,10 @@ static void xmlparse_dealloc(xmlparseobject *self) { PyObject_GC_UnTrack(self); + (void)xmlparse_clear(self); if (self->itself != NULL) XML_ParserFree(self->itself); self->itself = NULL; - (void)xmlparse_clear(self); if (self->handlers != NULL) { PyMem_Free(self->handlers); From webhook-mailer at python.org Sat Oct 2 10:15:54 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 02 Oct 2021 14:15:54 -0000 Subject: [Python-checkins] bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (GH-28687) (GH-28689) Message-ID: https://github.com/python/cpython/commit/d211e87307bb2a0b80e0a489501e892e61d879fc commit: d211e87307bb2a0b80e0a489501e892e61d879fc branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: terryjreedy date: 2021-10-02T10:15:45-04:00 summary: bpo-45341: Replace 'Packaging' with 'Package' in "Python P... Index" (GH-28687) (GH-28689) pypi.org " The Python Package Index (PyPI) ... (cherry picked from commit 0be338199fd663f020d833a4db185d0c5a0e0078) Co-authored-by: Terry Jan Reedy files: M Doc/distributing/index.rst M Doc/installing/index.rst diff --git a/Doc/distributing/index.rst b/Doc/distributing/index.rst index 66ba1e9fb9b34..136cf4e77b154 100644 --- a/Doc/distributing/index.rst +++ b/Doc/distributing/index.rst @@ -31,7 +31,7 @@ installing other Python projects, refer to the Key terms ========= -* the `Python Packaging Index `__ is a public +* the `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users * the `Python Packaging Authority @@ -127,14 +127,14 @@ involved in creating and publishing a project: * `Project structure`_ * `Building and packaging the project`_ -* `Uploading the project to the Python Packaging Index`_ +* `Uploading the project to the Python Package Index`_ * `The .pypirc file`_ .. _Project structure: \ https://packaging.python.org/tutorials/packaging-projects/#packaging-python-projects .. _Building and packaging the project: \ https://packaging.python.org/tutorials/packaging-projects/#creating-the-package-files -.. _Uploading the project to the Python Packaging Index: \ +.. _Uploading the project to the Python Package Index: \ https://packaging.python.org/tutorials/packaging-projects/#uploading-the-distribution-archives .. _The .pypirc file: \ https://packaging.python.org/specifications/pypirc/ @@ -150,7 +150,7 @@ These are quick answers or links for some common tasks. This isn't an easy topic, but here are a few tips: -* check the Python Packaging Index to see if the name is already in use +* check the Python Package Index to see if the name is already in use * check popular hosting sites like GitHub, Bitbucket, etc to see if there is already a project with that name * check what comes up in a web search for the name you're considering diff --git a/Doc/installing/index.rst b/Doc/installing/index.rst index 5e7e03045b2ac..4bacc7ba0c2cf 100644 --- a/Doc/installing/index.rst +++ b/Doc/installing/index.rst @@ -44,7 +44,7 @@ Key terms ``venv``. It allows virtual environments to be used on versions of Python prior to 3.4, which either don't provide ``venv`` at all, or aren't able to automatically install ``pip`` into created environments. -* The `Python Packaging Index `__ is a public +* The `Python Package Index `__ is a public repository of open source licensed packages made available for use by other Python users. * the `Python Packaging Authority @@ -78,7 +78,7 @@ The standard packaging tools are all designed to be used from the command line. The following command will install the latest version of a module and its -dependencies from the Python Packaging Index:: +dependencies from the Python Package Index:: python -m pip install SomePackage @@ -226,7 +226,7 @@ the installation process. With the introduction of support for the binary ``wheel`` format, and the ability to publish wheels for at least Windows and macOS through the -Python Packaging Index, this problem is expected to diminish over time, +Python Package Index, this problem is expected to diminish over time, as users are more regularly able to install pre-built extensions rather than needing to build them themselves. From webhook-mailer at python.org Sat Oct 2 14:48:20 2021 From: webhook-mailer at python.org (rhettinger) Date: Sat, 02 Oct 2021 18:48:20 -0000 Subject: [Python-checkins] bpo-45346: Keep docs consistent regarding true and false values (GH-28697) Message-ID: https://github.com/python/cpython/commit/db91b058d5d4fbff4185982095d90fe6a2741aed commit: db91b058d5d4fbff4185982095d90fe6a2741aed branch: main author: Raymond Hettinger committer: rhettinger date: 2021-10-02T13:48:08-05:00 summary: bpo-45346: Keep docs consistent regarding true and false values (GH-28697) files: M Doc/library/ast.rst M Doc/reference/compound_stmts.rst M Lib/test/test_builtin.py diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index e21151bd4ef79..d84c841fa4a08 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1266,7 +1266,7 @@ Pattern matching the pattern matches the subject. ``body`` contains a list of nodes to execute if the pattern matches and - the result of evaluating the guard expression is truthy. + the result of evaluating the guard expression is true. .. doctest:: diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index f24222f5fd9e9..5936cdf5ffc30 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -585,8 +585,8 @@ Here's an overview of the logical flow of a match statement: #. If the pattern succeeds, the corresponding guard (if present) is evaluated. In this case all name bindings are guaranteed to have happened. - * If the guard evaluates as truthy or missing, the ``block`` inside ``case_block`` is - executed. + * If the guard evaluates as true or is missing, the ``block`` inside + ``case_block`` is executed. * Otherwise, the next ``case_block`` is attempted as described above. @@ -637,10 +637,10 @@ The logical flow of a ``case`` block with a ``guard`` follows: #. If the pattern succeeded, evaluate the ``guard``. - * If the ``guard`` condition evaluates to "truthy", the case block is + * If the ``guard`` condition evaluates as true, the case block is selected. - * If the ``guard`` condition evaluates to "falsy", the case block is not + * If the ``guard`` condition evaluates as false, the case block is not selected. * If the ``guard`` raises an exception during evaluation, the exception diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index bd8353d038b6f..6dc4fa555021c 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1861,7 +1861,7 @@ def test_warning_notimplemented(self): # be evaluated in a boolean context (virtually all such use cases # are a result of accidental misuse implementing rich comparison # operations in terms of one another). - # For the time being, it will continue to evaluate as truthy, but + # For the time being, it will continue to evaluate as a true value, but # issue a deprecation warning (with the eventual intent to make it # a TypeError). self.assertWarns(DeprecationWarning, bool, NotImplemented) From webhook-mailer at python.org Sat Oct 2 14:52:12 2021 From: webhook-mailer at python.org (rhettinger) Date: Sat, 02 Oct 2021 18:52:12 -0000 Subject: [Python-checkins] Fix spelling error in comment (GH-28696) Message-ID: https://github.com/python/cpython/commit/dc878240dcbb47e660f5ad094deba5381872f2c9 commit: dc878240dcbb47e660f5ad094deba5381872f2c9 branch: main author: Raymond Hettinger committer: rhettinger date: 2021-10-02T13:52:05-05:00 summary: Fix spelling error in comment (GH-28696) files: M Objects/setobject.c diff --git a/Objects/setobject.c b/Objects/setobject.c index a3cdd33664d08..f71417d9f6ded 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -16,7 +16,7 @@ reduces the cost of hash collisions because consecutive memory accesses tend to be much cheaper than scattered probes. After LINEAR_PROBES steps, we then use more of the upper bits from the hash value and apply a simple - linear congruential random number genearator. This helps break-up long + linear congruential random number generator. This helps break-up long chains of collisions. All arithmetic on hash should ignore overflow. From webhook-mailer at python.org Sat Oct 2 15:33:03 2021 From: webhook-mailer at python.org (rhettinger) Date: Sat, 02 Oct 2021 19:33:03 -0000 Subject: [Python-checkins] bpo-45346: Keep docs consistent regarding true and false values (GH-28697) (GH-28698) Message-ID: https://github.com/python/cpython/commit/72089f33c0aed391f047b1cbaf19d8da1e51c167 commit: 72089f33c0aed391f047b1cbaf19d8da1e51c167 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: rhettinger date: 2021-10-02T14:32:56-05:00 summary: bpo-45346: Keep docs consistent regarding true and false values (GH-28697) (GH-28698) files: M Doc/library/ast.rst M Doc/reference/compound_stmts.rst M Lib/test/test_builtin.py diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index e21151bd4ef79..d84c841fa4a08 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1266,7 +1266,7 @@ Pattern matching the pattern matches the subject. ``body`` contains a list of nodes to execute if the pattern matches and - the result of evaluating the guard expression is truthy. + the result of evaluating the guard expression is true. .. doctest:: diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 25abc1be6a50a..41719be3dc986 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -585,8 +585,8 @@ Here's an overview of the logical flow of a match statement: #. If the pattern succeeds, the corresponding guard (if present) is evaluated. In this case all name bindings are guaranteed to have happened. - * If the guard evaluates as truthy or missing, the ``block`` inside ``case_block`` is - executed. + * If the guard evaluates as true or is missing, the ``block`` inside + ``case_block`` is executed. * Otherwise, the next ``case_block`` is attempted as described above. @@ -637,10 +637,10 @@ The logical flow of a ``case`` block with a ``guard`` follows: #. If the pattern succeeded, evaluate the ``guard``. - * If the ``guard`` condition evaluates to "truthy", the case block is + * If the ``guard`` condition evaluates as true, the case block is selected. - * If the ``guard`` condition evaluates to "falsy", the case block is not + * If the ``guard`` condition evaluates as false, the case block is not selected. * If the ``guard`` raises an exception during evaluation, the exception diff --git a/Lib/test/test_builtin.py b/Lib/test/test_builtin.py index bd8353d038b6f..6dc4fa555021c 100644 --- a/Lib/test/test_builtin.py +++ b/Lib/test/test_builtin.py @@ -1861,7 +1861,7 @@ def test_warning_notimplemented(self): # be evaluated in a boolean context (virtually all such use cases # are a result of accidental misuse implementing rich comparison # operations in terms of one another). - # For the time being, it will continue to evaluate as truthy, but + # For the time being, it will continue to evaluate as a true value, but # issue a deprecation warning (with the eventual intent to make it # a TypeError). self.assertWarns(DeprecationWarning, bool, NotImplemented) From webhook-mailer at python.org Sat Oct 2 15:33:58 2021 From: webhook-mailer at python.org (rhettinger) Date: Sat, 02 Oct 2021 19:33:58 -0000 Subject: [Python-checkins] Fix spelling error in comment (GH-28696) (GH-28699) Message-ID: https://github.com/python/cpython/commit/5ba61f488da2e092a81c3d126c5827dad6ab2006 commit: 5ba61f488da2e092a81c3d126c5827dad6ab2006 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: rhettinger date: 2021-10-02T14:33:49-05:00 summary: Fix spelling error in comment (GH-28696) (GH-28699) files: M Objects/setobject.c diff --git a/Objects/setobject.c b/Objects/setobject.c index caff85c9e3893..9d4cfd35e328f 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -16,7 +16,7 @@ reduces the cost of hash collisions because consecutive memory accesses tend to be much cheaper than scattered probes. After LINEAR_PROBES steps, we then use more of the upper bits from the hash value and apply a simple - linear congruential random number genearator. This helps break-up long + linear congruential random number generator. This helps break-up long chains of collisions. All arithmetic on hash should ignore overflow. From webhook-mailer at python.org Sun Oct 3 09:58:22 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 03 Oct 2021 13:58:22 -0000 Subject: [Python-checkins] Remove trailing spaces. (GH-28706) Message-ID: https://github.com/python/cpython/commit/a5a56154f14f3f4656a510e8b79e96d06289e654 commit: a5a56154f14f3f4656a510e8b79e96d06289e654 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-03T16:58:14+03:00 summary: Remove trailing spaces. (GH-28706) files: M Lib/test/test_syntax.py M Lib/test/test_tokenize.py M Modules/termios.c M Python/Python-tokenize.c M Python/bltinmodule.c diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index f4a507e91faa2..3ce362788da6e 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -1299,7 +1299,7 @@ def _check_error(self, code, errtext, self.assertEqual(err.end_lineno, end_lineno) if end_offset is not None: self.assertEqual(err.end_offset, end_offset) - + else: self.fail("compile() did not raise SyntaxError") @@ -1439,7 +1439,7 @@ def test_kwargs_last3(self): self._check_error("int(**{'base': 10}, *['2'])", "iterable argument unpacking follows " "keyword argument unpacking") - + def test_generator_in_function_call(self): self._check_error("foo(x, y for y in range(3) for z in range(2) if z , p)", "Generator expression must be parenthesized", diff --git a/Lib/test/test_tokenize.py b/Lib/test/test_tokenize.py index f8b16e5297645..ca2821de7c081 100644 --- a/Lib/test/test_tokenize.py +++ b/Lib/test/test_tokenize.py @@ -1933,7 +1933,7 @@ def test_string(self): c"""', """\ STRING 'rb"\""a\\\\\\nb\\\\\\nc"\""' (1, 0) (3, 4) """) - + self.check_tokenize('f"abc"', """\ STRING 'f"abc"' (1, 0) (1, 6) """) diff --git a/Modules/termios.c b/Modules/termios.c index 38573e25f51dd..354e5ca18d04d 100644 --- a/Modules/termios.c +++ b/Modules/termios.c @@ -408,7 +408,7 @@ termios_tcsetwinsize_impl(PyObject *module, int fd, PyObject *winsz) } Py_XDECREF(tmp_item); tmp_item = PySequence_GetItem(winsz, 1); - winsz_1 = PyLong_AsLong(tmp_item); + winsz_1 = PyLong_AsLong(tmp_item); if (winsz_1 == -1 && PyErr_Occurred()) { Py_XDECREF(tmp_item); return NULL; diff --git a/Python/Python-tokenize.c b/Python/Python-tokenize.c index 2933b5b7b1e20..fa713282558b0 100644 --- a/Python/Python-tokenize.c +++ b/Python/Python-tokenize.c @@ -150,7 +150,7 @@ static PyMethodDef tokenize_methods[] = { }; static PyModuleDef_Slot tokenizemodule_slots[] = { - {Py_mod_exec, tokenizemodule_exec}, + {Py_mod_exec, tokenizemodule_exec}, {0, NULL} }; diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index d0d31805b3018..1f038166890fd 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -2485,7 +2485,7 @@ builtin_sum_impl(PyObject *module, PyObject *iterable, PyObject *start) switch (Py_SIZE(item)) { case -1: b = -(sdigit) ((PyLongObject*)item)->ob_digit[0]; break; // Note: the continue goes to the top of the "while" loop that iterates over the elements - case 0: Py_DECREF(item); continue; + case 0: Py_DECREF(item); continue; case 1: b = ((PyLongObject*)item)->ob_digit[0]; break; default: b = PyLong_AsLongAndOverflow(item, &overflow); break; } From webhook-mailer at python.org Sun Oct 3 10:21:42 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 03 Oct 2021 14:21:42 -0000 Subject: [Python-checkins] Remove news entry without bpo issue number. (GH-28703) Message-ID: https://github.com/python/cpython/commit/4f6e0680d0d8545aa151ccd9de56a39bfe9532a2 commit: 4f6e0680d0d8545aa151ccd9de56a39bfe9532a2 branch: main author: Julien Palard committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-03T07:21:31-07:00 summary: Remove news entry without bpo issue number. (GH-28703) I'm just removing an erroneous NEWS entry I previously merged. Automerge-Triggered-By: GH:JulienPalard files: D Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst diff --git a/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst b/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst deleted file mode 100644 index a0ab4baf7952b3..00000000000000 --- a/Misc/NEWS.d/next/Build/2021-10-01-12-20-05.bpo-0.2ykYK2.rst +++ /dev/null @@ -1 +0,0 @@ -Makefile: fix missing slashes in some invocations cleaning previous build results when builing a macOS universal binary. \ No newline at end of file From webhook-mailer at python.org Sun Oct 3 13:03:58 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 03 Oct 2021 17:03:58 -0000 Subject: [Python-checkins] [3.10] Remove trailing spaces (GH-28709) Message-ID: https://github.com/python/cpython/commit/93242d7a2ad8c22f72ff84b63ed9046d32f6aa8e commit: 93242d7a2ad8c22f72ff84b63ed9046d32f6aa8e branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-03T20:03:49+03:00 summary: [3.10] Remove trailing spaces (GH-28709) files: M Include/internal/pycore_code.h M Lib/test/test_syntax.py M Lib/test/test_time.py M Lib/typing.py M Modules/_bisectmodule.c M Modules/_ctypes/_ctypes_test.c M Modules/_json.c M Modules/termios.c M Objects/exceptions.c M Objects/genericaliasobject.c M Objects/obmalloc.c M Parser/tokenizer.h M Python/bootstrap_hash.c M Tools/c-analyzer/c_parser/_state_machine.py M Tools/c-analyzer/c_parser/preprocessor/__init__.py diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index f1e89d96b9ebb..8ff1863dc0015 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -3,7 +3,7 @@ #ifdef __cplusplus extern "C" { #endif - + typedef struct { PyObject *ptr; /* Cached pointer (borrowed reference) */ uint64_t globals_ver; /* ma_version of global dict */ diff --git a/Lib/test/test_syntax.py b/Lib/test/test_syntax.py index f9deb7b3313a7..45b2785f34831 100644 --- a/Lib/test/test_syntax.py +++ b/Lib/test/test_syntax.py @@ -1298,7 +1298,7 @@ def _check_error(self, code, errtext, self.assertEqual(err.end_lineno, end_lineno) if end_offset is not None: self.assertEqual(err.end_offset, end_offset) - + else: self.fail("compile() did not raise SyntaxError") @@ -1438,7 +1438,7 @@ def test_kwargs_last3(self): self._check_error("int(**{'base': 10}, *['2'])", "iterable argument unpacking follows " "keyword argument unpacking") - + def test_generator_in_function_call(self): self._check_error("foo(x, y for y in range(3) for z in range(2) if z , p)", "Generator expression must be parenthesized", diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index 325829864851c..db929bd881778 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -1062,7 +1062,7 @@ def test_clock_functions(self): clock_names = [ "CLOCK_MONOTONIC", "clock_gettime", "clock_gettime_ns", "clock_settime", "clock_settime_ns", "clock_getres"] - + if mac_ver >= (10, 12): for name in clock_names: self.assertTrue(hasattr(time, name), f"time.{name} is not available") diff --git a/Lib/typing.py b/Lib/typing.py index 85908304270c0..f842fc23da6c6 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -99,7 +99,7 @@ 'NamedTuple', # Not really a type. 'TypedDict', # Not really a type. 'Generator', - + # Other concrete types. 'BinaryIO', 'IO', diff --git a/Modules/_bisectmodule.c b/Modules/_bisectmodule.c index aa63b685609cc..26c4b9bfb26b2 100644 --- a/Modules/_bisectmodule.c +++ b/Modules/_bisectmodule.c @@ -240,7 +240,7 @@ _bisect_insort_left_impl(PyObject *module, PyObject *a, PyObject *x, { PyObject *result, *key_x; Py_ssize_t index; - + if (key == Py_None) { index = internal_bisect_left(a, x, lo, hi, key); } else { diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c index 1ccad8e0e3d64..a33d15de9c0d4 100644 --- a/Modules/_ctypes/_ctypes_test.c +++ b/Modules/_ctypes/_ctypes_test.c @@ -1034,7 +1034,7 @@ EXPORT (HRESULT) KeepObject(IUnknown *punk) static struct PyModuleDef_Slot _ctypes_test_slots[] = { {0, NULL} -}; +}; static struct PyModuleDef _ctypes_testmodule = { PyModuleDef_HEAD_INIT, diff --git a/Modules/_json.c b/Modules/_json.c index e10f83c96c565..6f68c1f7f9b71 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -321,7 +321,7 @@ raise_errmsg(const char *msg, PyObject *s, Py_ssize_t end) if (decoder == NULL) { return; } - + _Py_IDENTIFIER(JSONDecodeError); PyObject *JSONDecodeError = _PyObject_GetAttrId(decoder, &PyId_JSONDecodeError); Py_DECREF(decoder); diff --git a/Modules/termios.c b/Modules/termios.c index a6649598ec171..fdfe589eb80c1 100644 --- a/Modules/termios.c +++ b/Modules/termios.c @@ -1004,7 +1004,7 @@ static void termiosmodule_free(void *m) { termiosmodule_clear((PyObject *)m); } -static int +static int termios_exec(PyObject *mod) { struct constant *constant = termios_constants; diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 38e523a802c09..6537a7ccd1e3c 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -1503,7 +1503,7 @@ SyntaxError_init(PySyntaxErrorObject *self, PyObject *args, PyObject *kwds) &self->end_lineno, &self->end_offset)) { Py_DECREF(info); return -1; - } + } Py_INCREF(self->filename); Py_INCREF(self->lineno); diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index e36e0c03476b5..dbe5d89b73962 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -578,7 +578,7 @@ static PyGetSetDef ga_properties[] = { }; /* A helper function to create GenericAlias' args tuple and set its attributes. - * Returns 1 on success, 0 on failure. + * Returns 1 on success, 0 on failure. */ static inline int setup_ga(gaobject *alias, PyObject *origin, PyObject *args) { diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index c1c12797aba11..1e06bee5c50ff 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -3035,7 +3035,7 @@ _PyObject_DebugMallocStats(FILE *out) fputc('\n', out); - /* Account for what all of those arena bytes are being used for. */ + /* Account for what all of those arena bytes are being used for. */ total = printone(out, "# bytes in allocated blocks", allocated_bytes); total += printone(out, "# bytes in available blocks", available_bytes); diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h index ff563d57fa8b1..a40f7d9687b44 100644 --- a/Parser/tokenizer.h +++ b/Parser/tokenizer.h @@ -83,7 +83,7 @@ struct tok_state { int async_def_nl; /* =1 if the outermost 'async def' had at least one NEWLINE token after it. */ /* How to proceed when asked for a new token in interactive mode */ - enum interactive_underflow_t interactive_underflow; + enum interactive_underflow_t interactive_underflow; }; extern struct tok_state *PyTokenizer_FromString(const char *, int); diff --git a/Python/bootstrap_hash.c b/Python/bootstrap_hash.c index a212f69870ed1..e189ce0d9014b 100644 --- a/Python/bootstrap_hash.c +++ b/Python/bootstrap_hash.c @@ -25,7 +25,7 @@ # include #endif -#if defined(__APPLE__) && defined(__has_builtin) +#if defined(__APPLE__) && defined(__has_builtin) # if __has_builtin(__builtin_available) # define HAVE_GETENTRYPY_GETRANDOM_RUNTIME __builtin_available(macOS 10.12, iOS 10.10, tvOS 10.0, watchOS 3.0, *) # endif @@ -221,7 +221,7 @@ py_getrandom(void *buffer, Py_ssize_t size, int blocking, int raise) #if defined(__APPLE__) && defined(__has_attribute) && __has_attribute(availability) static int -py_getentropy(char *buffer, Py_ssize_t size, int raise) +py_getentropy(char *buffer, Py_ssize_t size, int raise) __attribute__((availability(macos,introduced=10.12))) __attribute__((availability(ios,introduced=10.0))) __attribute__((availability(tvos,introduced=10.0))) diff --git a/Tools/c-analyzer/c_parser/_state_machine.py b/Tools/c-analyzer/c_parser/_state_machine.py index b505b4e3e4724..53cbb13e7c4ed 100644 --- a/Tools/c-analyzer/c_parser/_state_machine.py +++ b/Tools/c-analyzer/c_parser/_state_machine.py @@ -23,7 +23,7 @@ def parse(srclines): if isinstance(srclines, str): # a filename raise NotImplementedError - + # This only handles at most 10 nested levels. #MATCHED_PARENS = textwrap.dedent(rf''' diff --git a/Tools/c-analyzer/c_parser/preprocessor/__init__.py b/Tools/c-analyzer/c_parser/preprocessor/__init__.py index f206f694db5a8..8da4d8cadf7d0 100644 --- a/Tools/c-analyzer/c_parser/preprocessor/__init__.py +++ b/Tools/c-analyzer/c_parser/preprocessor/__init__.py @@ -91,7 +91,7 @@ def get_file_preprocessor(filename): macros = list(_resolve_file_values(filename, file_macros)) if file_incldirs: incldirs = [v for v, in _resolve_file_values(filename, file_incldirs)] - + def preprocess(**kwargs): if file_macros and 'macros' not in kwargs: kwargs['macros'] = macros From webhook-mailer at python.org Sun Oct 3 13:04:49 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 03 Oct 2021 17:04:49 -0000 Subject: [Python-checkins] [3.9] Remove trailing spaces (GH-28710) Message-ID: https://github.com/python/cpython/commit/e9ce081ec7fe6f45059e1de93952ad53e9c3aa74 commit: e9ce081ec7fe6f45059e1de93952ad53e9c3aa74 branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-03T20:04:38+03:00 summary: [3.9] Remove trailing spaces (GH-28710) files: M Include/internal/pycore_code.h M Lib/_osx_support.py M Modules/_ctypes/_ctypes_test.c M Modules/_ctypes/callproc.c M Modules/timemodule.c M Objects/genericaliasobject.c M Python/bootstrap_hash.c M Python/pytime.c diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 88956f109b4f7..a1bd6a0bc0f34 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -3,7 +3,7 @@ #ifdef __cplusplus extern "C" { #endif - + typedef struct { PyObject *ptr; /* Cached pointer (borrowed reference) */ uint64_t globals_ver; /* ma_version of global dict */ diff --git a/Lib/_osx_support.py b/Lib/_osx_support.py index 940c462e99952..2fc324a1efdd1 100644 --- a/Lib/_osx_support.py +++ b/Lib/_osx_support.py @@ -156,9 +156,9 @@ def _default_sysroot(cc): if _cache_default_sysroot is not None: return _cache_default_sysroot - + contents = _read_output('%s -c -E -v - "): in_incdirs = True diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c index 1ccad8e0e3d64..a33d15de9c0d4 100644 --- a/Modules/_ctypes/_ctypes_test.c +++ b/Modules/_ctypes/_ctypes_test.c @@ -1034,7 +1034,7 @@ EXPORT (HRESULT) KeepObject(IUnknown *punk) static struct PyModuleDef_Slot _ctypes_test_slots[] = { {0, NULL} -}; +}; static struct PyModuleDef _ctypes_testmodule = { PyModuleDef_HEAD_INIT, diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index b0f1e0bd0409b..dafc51e5d3db7 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -1484,14 +1484,14 @@ static PyObject *py_dyld_shared_cache_contains_path(PyObject *self, PyObject *ar if (!PyArg_ParseTuple(args, "O", &name)) return NULL; - + if (name == Py_None) Py_RETURN_FALSE; - + if (PyUnicode_FSConverter(name, &name2) == 0) return NULL; name_str = PyBytes_AS_STRING(name2); - + r = _dyld_shared_cache_contains_path(name_str); Py_DECREF(name2); diff --git a/Modules/timemodule.c b/Modules/timemodule.c index 80eab30c95d6f..df59f2aac5af2 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -51,7 +51,7 @@ #define _Py_tzname tzname #endif -#if defined(__APPLE__ ) && defined(__has_builtin) +#if defined(__APPLE__ ) && defined(__has_builtin) # if __has_builtin(__builtin_available) # define HAVE_CLOCK_GETTIME_RUNTIME __builtin_available(macOS 10.12, iOS 10.0, tvOS 10.0, watchOS 3.0, *) # endif @@ -160,7 +160,7 @@ perf_counter(_Py_clock_info_t *info) #ifdef HAVE_CLOCK_GETTIME #ifdef __APPLE__ -/* +/* * The clock_* functions will be removed from the module * dict entirely when the C API is not available. */ @@ -1421,7 +1421,7 @@ _PyTime_GetThreadTimeWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) #if defined(__APPLE__) && defined(__has_attribute) && __has_attribute(availability) static int -_PyTime_GetThreadTimeWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) +_PyTime_GetThreadTimeWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) __attribute__((availability(macos, introduced=10.12))) __attribute__((availability(ios, introduced=10.0))) __attribute__((availability(tvos, introduced=10.0))) @@ -1460,7 +1460,7 @@ _PyTime_GetThreadTimeWithInfo(_PyTime_t *tp, _Py_clock_info_t *info) #ifdef HAVE_THREAD_TIME #ifdef __APPLE__ -/* +/* * The clock_* functions will be removed from the module * dict entirely when the C API is not available. */ diff --git a/Objects/genericaliasobject.c b/Objects/genericaliasobject.c index 4ab762c00e2a6..acbb01cfef92c 100644 --- a/Objects/genericaliasobject.c +++ b/Objects/genericaliasobject.c @@ -566,7 +566,7 @@ static PyGetSetDef ga_properties[] = { }; /* A helper function to create GenericAlias' args tuple and set its attributes. - * Returns 1 on success, 0 on failure. + * Returns 1 on success, 0 on failure. */ static inline int setup_ga(gaobject *alias, PyObject *origin, PyObject *args) { diff --git a/Python/bootstrap_hash.c b/Python/bootstrap_hash.c index a212f69870ed1..e189ce0d9014b 100644 --- a/Python/bootstrap_hash.c +++ b/Python/bootstrap_hash.c @@ -25,7 +25,7 @@ # include #endif -#if defined(__APPLE__) && defined(__has_builtin) +#if defined(__APPLE__) && defined(__has_builtin) # if __has_builtin(__builtin_available) # define HAVE_GETENTRYPY_GETRANDOM_RUNTIME __builtin_available(macOS 10.12, iOS 10.10, tvOS 10.0, watchOS 3.0, *) # endif @@ -221,7 +221,7 @@ py_getrandom(void *buffer, Py_ssize_t size, int blocking, int raise) #if defined(__APPLE__) && defined(__has_attribute) && __has_attribute(availability) static int -py_getentropy(char *buffer, Py_ssize_t size, int raise) +py_getentropy(char *buffer, Py_ssize_t size, int raise) __attribute__((availability(macos,introduced=10.12))) __attribute__((availability(ios,introduced=10.0))) __attribute__((availability(tvos,introduced=10.0))) diff --git a/Python/pytime.c b/Python/pytime.c index 89d63e080422b..a9af757243127 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -6,7 +6,7 @@ #if defined(__APPLE__) #include /* mach_absolute_time(), mach_timebase_info() */ -#if defined(__APPLE__) && defined(__has_builtin) +#if defined(__APPLE__) && defined(__has_builtin) # if __has_builtin(__builtin_available) # define HAVE_CLOCK_GETTIME_RUNTIME __builtin_available(macOS 10.12, iOS 10.0, tvOS 10.0, watchOS 3.0, *) # endif @@ -730,7 +730,7 @@ pygettimeofday(_PyTime_t *tp, _Py_clock_info_t *info, int raise) } #ifdef HAVE_CLOCK_GETTIME_RUNTIME - } else { + } else { #endif #endif From webhook-mailer at python.org Sun Oct 3 14:22:54 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 03 Oct 2021 18:22:54 -0000 Subject: [Python-checkins] bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711) Message-ID: https://github.com/python/cpython/commit/60b9e040c9cf40e69f42c0008e564458aa0379e8 commit: 60b9e040c9cf40e69f42c0008e564458aa0379e8 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-03T21:22:42+03:00 summary: bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711) files: M Modules/_tracemalloc.c M Objects/frameobject.c M Objects/genobject.c M Python/ceval.c M Python/traceback.c diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index fc3d7f51ee29a..09d18fb8f278f 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -302,7 +302,7 @@ static void tracemalloc_get_frame(InterpreterFrame *pyframe, frame_t *frame) { frame->filename = unknown_filename; - int lineno = PyCode_Addr2Line(pyframe->f_code, pyframe->f_lasti*2); + int lineno = PyCode_Addr2Line(pyframe->f_code, pyframe->f_lasti*sizeof(_Py_CODEUNIT)); if (lineno < 0) { lineno = 0; } diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 00d6888ff2a2a..b743dc72eee79 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -45,7 +45,7 @@ PyFrame_GetLineNumber(PyFrameObject *f) return f->f_lineno; } else { - return PyCode_Addr2Line(f->f_frame->f_code, f->f_frame->f_lasti*2); + return PyCode_Addr2Line(f->f_frame->f_code, f->f_frame->f_lasti*sizeof(_Py_CODEUNIT)); } } @@ -67,7 +67,7 @@ frame_getlasti(PyFrameObject *f, void *closure) if (f->f_frame->f_lasti < 0) { return PyLong_FromLong(-1); } - return PyLong_FromLong(f->f_frame->f_lasti*2); + return PyLong_FromLong(f->f_frame->f_lasti*sizeof(_Py_CODEUNIT)); } static PyObject * diff --git a/Objects/genobject.c b/Objects/genobject.c index be9238d9b6cfd..8bd6c8d2c4ccc 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1284,7 +1284,7 @@ compute_cr_origin(int origin_depth) PyCodeObject *code = frame->f_code; PyObject *frameinfo = Py_BuildValue("OiO", code->co_filename, - PyCode_Addr2Line(frame->f_code, frame->f_lasti*2), + PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)), code->co_name); if (!frameinfo) { Py_DECREF(cr_origin); diff --git a/Python/ceval.c b/Python/ceval.c index 7f29967eb3272..c951e563cd7a1 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4815,7 +4815,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr #endif fprintf(stderr, "XXX lineno: %d, opcode: %d\n", - PyCode_Addr2Line(frame->f_code, frame->f_lasti*2), + PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)), opcode); _PyErr_SetString(tstate, PyExc_SystemError, "unknown opcode"); goto error; @@ -5996,7 +5996,7 @@ call_trace(Py_tracefunc func, PyObject *obj, } else { initialize_trace_info(&tstate->trace_info, frame); - f->f_lineno = _PyCode_CheckLineNumber(frame->f_lasti*2, &tstate->trace_info.bounds); + f->f_lineno = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); } result = func(obj, f, what, arg); f->f_lineno = 0; @@ -6035,8 +6035,8 @@ maybe_call_line_trace(Py_tracefunc func, PyObject *obj, then call the trace function if we're tracing source lines. */ initialize_trace_info(&tstate->trace_info, frame); - int lastline = _PyCode_CheckLineNumber(instr_prev*2, &tstate->trace_info.bounds); - int line = _PyCode_CheckLineNumber(frame->f_lasti*2, &tstate->trace_info.bounds); + int lastline = _PyCode_CheckLineNumber(instr_prev*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); + int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &tstate->trace_info.bounds); PyFrameObject *f = _PyFrame_GetFrameObject(frame); if (f == NULL) { return -1; @@ -6978,7 +6978,7 @@ dtrace_function_entry(InterpreterFrame *frame) PyCodeObject *code = frame->f_code; filename = PyUnicode_AsUTF8(code->co_filename); funcname = PyUnicode_AsUTF8(code->co_name); - lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*2); + lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)); PyDTrace_FUNCTION_ENTRY(filename, funcname, lineno); } @@ -6993,7 +6993,7 @@ dtrace_function_return(InterpreterFrame *frame) PyCodeObject *code = frame->f_code; filename = PyUnicode_AsUTF8(code->co_filename); funcname = PyUnicode_AsUTF8(code->co_name); - lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*2); + lineno = PyCode_Addr2Line(frame->f_code, frame->f_lasti*sizeof(_Py_CODEUNIT)); PyDTrace_FUNCTION_RETURN(filename, funcname, lineno); } @@ -7010,12 +7010,12 @@ maybe_dtrace_line(InterpreterFrame *frame, instruction window, reset the window. */ initialize_trace_info(trace_info, frame); - int lastline = _PyCode_CheckLineNumber(instr_prev*2, &trace_info->bounds); - int line = _PyCode_CheckLineNumber(frame->f_lasti*2, &trace_info->bounds); + int lastline = _PyCode_CheckLineNumber(instr_prev*sizeof(_Py_CODEUNIT), &trace_info->bounds); + int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &trace_info->bounds); if (line != -1) { /* Trace backward edges or first instruction of a new line */ if (frame->f_lasti < instr_prev || - (line != lastline && frame->f_lasti*2 == trace_info->bounds.ar_start)) + (line != lastline && frame->f_lasti*sizeof(_Py_CODEUNIT) == trace_info->bounds.ar_start)) { co_filename = PyUnicode_AsUTF8(frame->f_code->co_filename); if (!co_filename) { diff --git a/Python/traceback.c b/Python/traceback.c index 76280a35e3a5f..06b40bbbdc9f8 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -240,7 +240,7 @@ _PyTraceBack_FromFrame(PyObject *tb_next, PyFrameObject *frame) assert(tb_next == NULL || PyTraceBack_Check(tb_next)); assert(frame != NULL); - return tb_create_raw((PyTracebackObject *)tb_next, frame, frame->f_frame->f_lasti*2, + return tb_create_raw((PyTracebackObject *)tb_next, frame, frame->f_frame->f_lasti*sizeof(_Py_CODEUNIT), PyFrame_GetLineNumber(frame)); } @@ -1047,7 +1047,7 @@ dump_frame(int fd, InterpreterFrame *frame) PUTS(fd, "???"); } - int lineno = PyCode_Addr2Line(code, frame->f_lasti*2); + int lineno = PyCode_Addr2Line(code, frame->f_lasti*sizeof(_Py_CODEUNIT)); PUTS(fd, ", line "); if (lineno >= 0) { _Py_DumpDecimal(fd, (size_t)lineno); From webhook-mailer at python.org Sun Oct 3 19:40:02 2021 From: webhook-mailer at python.org (pablogsal) Date: Sun, 03 Oct 2021 23:40:02 -0000 Subject: [Python-checkins] bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28707) Message-ID: https://github.com/python/cpython/commit/f146ca36f81075f222aa3a1595042597d96dfad3 commit: f146ca36f81075f222aa3a1595042597d96dfad3 branch: 3.10 author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-04T00:39:54+01:00 summary: bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28707) files: M .github/workflows/build.yml M aclocal.m4 M configure diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index 0607a565514bb..88f825e613bca 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -106,7 +106,9 @@ jobs: - name: Check limited ABI symbols run: make check-limited-abi - name: Check Autoconf version 2.69 - run: grep "Generated by GNU Autoconf 2.69" configure + run: | + grep "Generated by GNU Autoconf 2.69" configure + grep "PKG_PROG_PKG_CONFIG" aclocal.m4 build_win32: name: 'Windows (x86)' @@ -147,8 +149,12 @@ jobs: PYTHONSTRICTEXTENSIONBUILD: 1 steps: - uses: actions/checkout at v2 + - name: Prepare homebrew environment variables + run: | + echo "LDFLAGS=-L$(brew --prefix tcl-tk)/lib" >> $GITHUB_ENV + echo "PKG_CONFIG_PATH=$(brew --prefix openssl at 1.1)/lib/pkgconfig:$(brew --prefix tcl-tk)/lib/pkgconfig" >> $GITHUB_ENV - name: Configure CPython - run: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-dev + run: ./configure --with-pydebug --prefix=/opt/python-dev - name: Build CPython run: make -j4 - name: Display build info diff --git a/aclocal.m4 b/aclocal.m4 index 987bfdf215ccb..2f1bd37528c85 100644 --- a/aclocal.m4 +++ b/aclocal.m4 @@ -275,3 +275,347 @@ AC_DEFUN([AX_CHECK_OPENSSL], [ AC_SUBST([OPENSSL_LDFLAGS]) ]) +# pkg.m4 - Macros to locate and utilise pkg-config. -*- Autoconf -*- +# serial 11 (pkg-config-0.29.1) + +dnl Copyright ? 2004 Scott James Remnant . +dnl Copyright ? 2012-2015 Dan Nicholson +dnl +dnl This program is free software; you can redistribute it and/or modify +dnl it under the terms of the GNU General Public License as published by +dnl the Free Software Foundation; either version 2 of the License, or +dnl (at your option) any later version. +dnl +dnl This program is distributed in the hope that it will be useful, but +dnl WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl General Public License for more details. +dnl +dnl You should have received a copy of the GNU General Public License +dnl along with this program; if not, write to the Free Software +dnl Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA +dnl 02111-1307, USA. +dnl +dnl As a special exception to the GNU General Public License, if you +dnl distribute this file as part of a program that contains a +dnl configuration script generated by Autoconf, you may include it under +dnl the same distribution terms that you use for the rest of that +dnl program. + +dnl PKG_PREREQ(MIN-VERSION) +dnl ----------------------- +dnl Since: 0.29 +dnl +dnl Verify that the version of the pkg-config macros are at least +dnl MIN-VERSION. Unlike PKG_PROG_PKG_CONFIG, which checks the user's +dnl installed version of pkg-config, this checks the developer's version +dnl of pkg.m4 when generating configure. +dnl +dnl To ensure that this macro is defined, also add: +dnl m4_ifndef([PKG_PREREQ], +dnl [m4_fatal([must install pkg-config 0.29 or later before running autoconf/autogen])]) +dnl +dnl See the "Since" comment for each macro you use to see what version +dnl of the macros you require. +m4_defun([PKG_PREREQ], +[m4_define([PKG_MACROS_VERSION], [0.29.1]) +m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1, + [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])]) +])dnl PKG_PREREQ + +dnl PKG_PROG_PKG_CONFIG([MIN-VERSION]) +dnl ---------------------------------- +dnl Since: 0.16 +dnl +dnl Search for the pkg-config tool and set the PKG_CONFIG variable to +dnl first found in the path. Checks that the version of pkg-config found +dnl is at least MIN-VERSION. If MIN-VERSION is not specified, 0.9.0 is +dnl used since that's the first version where most current features of +dnl pkg-config existed. +AC_DEFUN([PKG_PROG_PKG_CONFIG], +[m4_pattern_forbid([^_?PKG_[A-Z_]+$]) +m4_pattern_allow([^PKG_CONFIG(_(PATH|LIBDIR|SYSROOT_DIR|ALLOW_SYSTEM_(CFLAGS|LIBS)))?$]) +m4_pattern_allow([^PKG_CONFIG_(DISABLE_UNINSTALLED|TOP_BUILD_DIR|DEBUG_SPEW)$]) +AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility]) +AC_ARG_VAR([PKG_CONFIG_PATH], [directories to add to pkg-config's search path]) +AC_ARG_VAR([PKG_CONFIG_LIBDIR], [path overriding pkg-config's built-in search path]) + +if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then + AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) +fi +if test -n "$PKG_CONFIG"; then + _pkg_min_version=m4_default([$1], [0.9.0]) + AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) + if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then + AC_MSG_RESULT([yes]) + else + AC_MSG_RESULT([no]) + PKG_CONFIG="" + fi +fi[]dnl +])dnl PKG_PROG_PKG_CONFIG + +dnl PKG_CHECK_EXISTS(MODULES, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------------------------------- +dnl Since: 0.18 +dnl +dnl Check to see whether a particular set of modules exists. Similar to +dnl PKG_CHECK_MODULES(), but does not set variables or print errors. +dnl +dnl Please remember that m4 expands AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +dnl only at the first occurence in configure.ac, so if the first place +dnl it's called might be skipped (such as if it is within an "if", you +dnl have to call PKG_CHECK_EXISTS manually +AC_DEFUN([PKG_CHECK_EXISTS], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +if test -n "$PKG_CONFIG" && \ + AC_RUN_LOG([$PKG_CONFIG --exists --print-errors "$1"]); then + m4_default([$2], [:]) +m4_ifvaln([$3], [else + $3])dnl +fi]) + +dnl _PKG_CONFIG([VARIABLE], [COMMAND], [MODULES]) +dnl --------------------------------------------- +dnl Internal wrapper calling pkg-config via PKG_CONFIG and setting +dnl pkg_failed based on the result. +m4_define([_PKG_CONFIG], +[if test -n "$$1"; then + pkg_cv_[]$1="$$1" + elif test -n "$PKG_CONFIG"; then + PKG_CHECK_EXISTS([$3], + [pkg_cv_[]$1=`$PKG_CONFIG --[]$2 "$3" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes ], + [pkg_failed=yes]) + else + pkg_failed=untried +fi[]dnl +])dnl _PKG_CONFIG + +dnl _PKG_SHORT_ERRORS_SUPPORTED +dnl --------------------------- +dnl Internal check to see if pkg-config supports short errors. +AC_DEFUN([_PKG_SHORT_ERRORS_SUPPORTED], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi[]dnl +])dnl _PKG_SHORT_ERRORS_SUPPORTED + + +dnl PKG_CHECK_MODULES(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl -------------------------------------------------------------- +dnl Since: 0.4.0 +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES might not happen, you should be sure to include an +dnl explicit call to PKG_PROG_PKG_CONFIG in your configure.ac +AC_DEFUN([PKG_CHECK_MODULES], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl +AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl + +pkg_failed=no +AC_MSG_CHECKING([for $1]) + +_PKG_CONFIG([$1][_CFLAGS], [cflags], [$2]) +_PKG_CONFIG([$1][_LIBS], [libs], [$2]) + +m4_define([_PKG_TEXT], [Alternatively, you may set the environment variables $1[]_CFLAGS +and $1[]_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details.]) + +if test $pkg_failed = yes; then + AC_MSG_RESULT([no]) + _PKG_SHORT_ERRORS_SUPPORTED + if test $_pkg_short_errors_supported = yes; then + $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1` + else + $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$$1[]_PKG_ERRORS" >&AS_MESSAGE_LOG_FD + + m4_default([$4], [AC_MSG_ERROR( +[Package requirements ($2) were not met: + +$$1_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +_PKG_TEXT])[]dnl + ]) +elif test $pkg_failed = untried; then + AC_MSG_RESULT([no]) + m4_default([$4], [AC_MSG_FAILURE( +[The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +_PKG_TEXT + +To get pkg-config, see .])[]dnl + ]) +else + $1[]_CFLAGS=$pkg_cv_[]$1[]_CFLAGS + $1[]_LIBS=$pkg_cv_[]$1[]_LIBS + AC_MSG_RESULT([yes]) + $3 +fi[]dnl +])dnl PKG_CHECK_MODULES + + +dnl PKG_CHECK_MODULES_STATIC(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl --------------------------------------------------------------------- +dnl Since: 0.29 +dnl +dnl Checks for existence of MODULES and gathers its build flags with +dnl static libraries enabled. Sets VARIABLE-PREFIX_CFLAGS from --cflags +dnl and VARIABLE-PREFIX_LIBS from --libs. +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES_STATIC might not happen, you should be sure to +dnl include an explicit call to PKG_PROG_PKG_CONFIG in your +dnl configure.ac. +AC_DEFUN([PKG_CHECK_MODULES_STATIC], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +_save_PKG_CONFIG=$PKG_CONFIG +PKG_CONFIG="$PKG_CONFIG --static" +PKG_CHECK_MODULES($@) +PKG_CONFIG=$_save_PKG_CONFIG[]dnl +])dnl PKG_CHECK_MODULES_STATIC + + +dnl PKG_INSTALLDIR([DIRECTORY]) +dnl ------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable pkgconfigdir as the location where a module +dnl should install pkg-config .pc files. By default the directory is +dnl $libdir/pkgconfig, but the default can be changed by passing +dnl DIRECTORY. The user can override through the --with-pkgconfigdir +dnl parameter. +AC_DEFUN([PKG_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${libdir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([pkgconfigdir], + [AS_HELP_STRING([--with-pkgconfigdir], pkg_description)],, + [with_pkgconfigdir=]pkg_default) +AC_SUBST([pkgconfigdir], [$with_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_INSTALLDIR + + +dnl PKG_NOARCH_INSTALLDIR([DIRECTORY]) +dnl -------------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable noarch_pkgconfigdir as the location where a +dnl module should install arch-independent pkg-config .pc files. By +dnl default the directory is $datadir/pkgconfig, but the default can be +dnl changed by passing DIRECTORY. The user can override through the +dnl --with-noarch-pkgconfigdir parameter. +AC_DEFUN([PKG_NOARCH_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${datadir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config arch-independent installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([noarch-pkgconfigdir], + [AS_HELP_STRING([--with-noarch-pkgconfigdir], pkg_description)],, + [with_noarch_pkgconfigdir=]pkg_default) +AC_SUBST([noarch_pkgconfigdir], [$with_noarch_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_NOARCH_INSTALLDIR + + +dnl PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE, +dnl [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------- +dnl Since: 0.28 +dnl +dnl Retrieves the value of the pkg-config variable for the given module. +AC_DEFUN([PKG_CHECK_VAR], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl + +_PKG_CONFIG([$1], [variable="][$3]["], [$2]) +AS_VAR_COPY([$1], [pkg_cv_][$1]) + +AS_VAR_IF([$1], [""], [$5], [$4])dnl +])dnl PKG_CHECK_VAR + +dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND], +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------ +dnl +dnl Prepare a "--with-" configure option using the lowercase +dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and +dnl PKG_CHECK_MODULES in a single macro. +AC_DEFUN([PKG_WITH_MODULES], +[ +m4_pushdef([with_arg], m4_tolower([$1])) + +m4_pushdef([description], + [m4_default([$5], [build with ]with_arg[ support])]) + +m4_pushdef([def_arg], [m4_default([$6], [auto])]) +m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes]) +m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no]) + +m4_case(def_arg, + [yes],[m4_pushdef([with_without], [--without-]with_arg)], + [m4_pushdef([with_without],[--with-]with_arg)]) + +AC_ARG_WITH(with_arg, + AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),, + [AS_TR_SH([with_]with_arg)=def_arg]) + +AS_CASE([$AS_TR_SH([with_]with_arg)], + [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)], + [auto],[PKG_CHECK_MODULES([$1],[$2], + [m4_n([def_action_if_found]) $3], + [m4_n([def_action_if_not_found]) $4])]) + +m4_popdef([with_arg]) +m4_popdef([description]) +m4_popdef([def_arg]) + +])dnl PKG_WITH_MODULES + +dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ----------------------------------------------- +dnl +dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES +dnl check._[VARIABLE-PREFIX] is exported as make variable. +AC_DEFUN([PKG_HAVE_WITH_MODULES], +[ +PKG_WITH_MODULES([$1],[$2],,,[$3],[$4]) + +AM_CONDITIONAL([HAVE_][$1], + [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"]) +])dnl PKG_HAVE_WITH_MODULES + +dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------------------ +dnl +dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after +dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make +dnl and preprocessor variable. +AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES], +[ +PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4]) + +AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"], + [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])]) +])dnl PKG_HAVE_DEFINE_WITH_MODULES + diff --git a/configure b/configure index 753f9564693bc..02d882ed39d68 100755 --- a/configure +++ b/configure @@ -630,7 +630,6 @@ OPENSSL_RPATH OPENSSL_LDFLAGS OPENSSL_LIBS OPENSSL_INCLUDES -PKG_CONFIG ENSUREPIP SRCDIRS THREADHEADERS @@ -662,6 +661,9 @@ DTRACE TCLTK_LIBS TCLTK_INCLUDES LIBFFI_INCLUDEDIR +PKG_CONFIG_LIBDIR +PKG_CONFIG_PATH +PKG_CONFIG TZPATH SHLIBS CFLAGSFORSHARED @@ -873,7 +875,10 @@ LDFLAGS LIBS CPPFLAGS CPP -PROFILE_TASK' +PROFILE_TASK +PKG_CONFIG +PKG_CONFIG_PATH +PKG_CONFIG_LIBDIR' # Initialize some variables set by options. @@ -1637,6 +1642,11 @@ Some influential environment variables: CPP C preprocessor PROFILE_TASK Python args for PGO generation task + PKG_CONFIG path to pkg-config utility + PKG_CONFIG_PATH + directories to add to pkg-config's search path + PKG_CONFIG_LIBDIR + path overriding pkg-config's built-in search path Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. @@ -10542,7 +10552,126 @@ $as_echo "no" >&6; } fi -PKG_PROG_PKG_CONFIG + + + + + + + +if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then + if test -n "$ac_tool_prefix"; then + # Extract the first word of "${ac_tool_prefix}pkg-config", so it can be a program name with args. +set dummy ${ac_tool_prefix}pkg-config; ac_word=$2 +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 +$as_echo_n "checking for $ac_word... " >&6; } +if ${ac_cv_path_PKG_CONFIG+:} false; then : + $as_echo_n "(cached) " >&6 +else + case $PKG_CONFIG in + [\\/]* | ?:[\\/]*) + ac_cv_path_PKG_CONFIG="$PKG_CONFIG" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" + $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done + done +IFS=$as_save_IFS + + ;; +esac +fi +PKG_CONFIG=$ac_cv_path_PKG_CONFIG +if test -n "$PKG_CONFIG"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PKG_CONFIG" >&5 +$as_echo "$PKG_CONFIG" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi + + +fi +if test -z "$ac_cv_path_PKG_CONFIG"; then + ac_pt_PKG_CONFIG=$PKG_CONFIG + # Extract the first word of "pkg-config", so it can be a program name with args. +set dummy pkg-config; ac_word=$2 +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 +$as_echo_n "checking for $ac_word... " >&6; } +if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then : + $as_echo_n "(cached) " >&6 +else + case $ac_pt_PKG_CONFIG in + [\\/]* | ?:[\\/]*) + ac_cv_path_ac_pt_PKG_CONFIG="$ac_pt_PKG_CONFIG" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_ac_pt_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" + $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done + done +IFS=$as_save_IFS + + ;; +esac +fi +ac_pt_PKG_CONFIG=$ac_cv_path_ac_pt_PKG_CONFIG +if test -n "$ac_pt_PKG_CONFIG"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_pt_PKG_CONFIG" >&5 +$as_echo "$ac_pt_PKG_CONFIG" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi + + if test "x$ac_pt_PKG_CONFIG" = x; then + PKG_CONFIG="" + else + case $cross_compiling:$ac_tool_warned in +yes:) +{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 +$as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} +ac_tool_warned=yes ;; +esac + PKG_CONFIG=$ac_pt_PKG_CONFIG + fi +else + PKG_CONFIG="$ac_cv_path_PKG_CONFIG" +fi + +fi +if test -n "$PKG_CONFIG"; then + _pkg_min_version=0.9.0 + { $as_echo "$as_me:${as_lineno-$LINENO}: checking pkg-config is at least version $_pkg_min_version" >&5 +$as_echo_n "checking pkg-config is at least version $_pkg_min_version... " >&6; } + if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + PKG_CONFIG="" + fi +fi # Check for use of the system expat library { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-system-expat" >&5 From webhook-mailer at python.org Sun Oct 3 19:47:00 2021 From: webhook-mailer at python.org (pablogsal) Date: Sun, 03 Oct 2021 23:47:00 -0000 Subject: [Python-checkins] bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28708) Message-ID: https://github.com/python/cpython/commit/a25dcaefb7c4eb0767a112cd31fe0b055f168844 commit: a25dcaefb7c4eb0767a112cd31fe0b055f168844 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-04T00:46:52+01:00 summary: bpo-45350: Rerun autoreconf with the pkg-config macros (GH-28708) files: M .github/workflows/build.yml M aclocal.m4 M configure diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index b0a18913bf6a0..ec3acf77563f7 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -85,7 +85,9 @@ jobs: - name: Check limited ABI symbols run: make check-limited-abi - name: Check Autoconf version 2.69 - run: grep "Generated by GNU Autoconf 2.69" configure + run: | + grep "Generated by GNU Autoconf 2.69" configure + grep "PKG_PROG_PKG_CONFIG" aclocal.m4 build_win32: name: 'Windows (x86)' @@ -126,8 +128,12 @@ jobs: PYTHONSTRICTEXTENSIONBUILD: 1 steps: - uses: actions/checkout at v2 + - name: Prepare homebrew environment variables + run: | + echo "LDFLAGS=-L$(brew --prefix tcl-tk)/lib" >> $GITHUB_ENV + echo "PKG_CONFIG_PATH=$(brew --prefix openssl at 1.1)/lib/pkgconfig:$(brew --prefix tcl-tk)/lib/pkgconfig" >> $GITHUB_ENV - name: Configure CPython - run: ./configure --with-pydebug --with-openssl=/usr/local/opt/openssl --prefix=/opt/python-dev + run: ./configure --with-pydebug --prefix=/opt/python-dev - name: Build CPython run: make -j4 - name: Display build info diff --git a/aclocal.m4 b/aclocal.m4 index 987bfdf215ccb..2f1bd37528c85 100644 --- a/aclocal.m4 +++ b/aclocal.m4 @@ -275,3 +275,347 @@ AC_DEFUN([AX_CHECK_OPENSSL], [ AC_SUBST([OPENSSL_LDFLAGS]) ]) +# pkg.m4 - Macros to locate and utilise pkg-config. -*- Autoconf -*- +# serial 11 (pkg-config-0.29.1) + +dnl Copyright ? 2004 Scott James Remnant . +dnl Copyright ? 2012-2015 Dan Nicholson +dnl +dnl This program is free software; you can redistribute it and/or modify +dnl it under the terms of the GNU General Public License as published by +dnl the Free Software Foundation; either version 2 of the License, or +dnl (at your option) any later version. +dnl +dnl This program is distributed in the hope that it will be useful, but +dnl WITHOUT ANY WARRANTY; without even the implied warranty of +dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +dnl General Public License for more details. +dnl +dnl You should have received a copy of the GNU General Public License +dnl along with this program; if not, write to the Free Software +dnl Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA +dnl 02111-1307, USA. +dnl +dnl As a special exception to the GNU General Public License, if you +dnl distribute this file as part of a program that contains a +dnl configuration script generated by Autoconf, you may include it under +dnl the same distribution terms that you use for the rest of that +dnl program. + +dnl PKG_PREREQ(MIN-VERSION) +dnl ----------------------- +dnl Since: 0.29 +dnl +dnl Verify that the version of the pkg-config macros are at least +dnl MIN-VERSION. Unlike PKG_PROG_PKG_CONFIG, which checks the user's +dnl installed version of pkg-config, this checks the developer's version +dnl of pkg.m4 when generating configure. +dnl +dnl To ensure that this macro is defined, also add: +dnl m4_ifndef([PKG_PREREQ], +dnl [m4_fatal([must install pkg-config 0.29 or later before running autoconf/autogen])]) +dnl +dnl See the "Since" comment for each macro you use to see what version +dnl of the macros you require. +m4_defun([PKG_PREREQ], +[m4_define([PKG_MACROS_VERSION], [0.29.1]) +m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1, + [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])]) +])dnl PKG_PREREQ + +dnl PKG_PROG_PKG_CONFIG([MIN-VERSION]) +dnl ---------------------------------- +dnl Since: 0.16 +dnl +dnl Search for the pkg-config tool and set the PKG_CONFIG variable to +dnl first found in the path. Checks that the version of pkg-config found +dnl is at least MIN-VERSION. If MIN-VERSION is not specified, 0.9.0 is +dnl used since that's the first version where most current features of +dnl pkg-config existed. +AC_DEFUN([PKG_PROG_PKG_CONFIG], +[m4_pattern_forbid([^_?PKG_[A-Z_]+$]) +m4_pattern_allow([^PKG_CONFIG(_(PATH|LIBDIR|SYSROOT_DIR|ALLOW_SYSTEM_(CFLAGS|LIBS)))?$]) +m4_pattern_allow([^PKG_CONFIG_(DISABLE_UNINSTALLED|TOP_BUILD_DIR|DEBUG_SPEW)$]) +AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility]) +AC_ARG_VAR([PKG_CONFIG_PATH], [directories to add to pkg-config's search path]) +AC_ARG_VAR([PKG_CONFIG_LIBDIR], [path overriding pkg-config's built-in search path]) + +if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then + AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) +fi +if test -n "$PKG_CONFIG"; then + _pkg_min_version=m4_default([$1], [0.9.0]) + AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) + if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then + AC_MSG_RESULT([yes]) + else + AC_MSG_RESULT([no]) + PKG_CONFIG="" + fi +fi[]dnl +])dnl PKG_PROG_PKG_CONFIG + +dnl PKG_CHECK_EXISTS(MODULES, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------------------------------- +dnl Since: 0.18 +dnl +dnl Check to see whether a particular set of modules exists. Similar to +dnl PKG_CHECK_MODULES(), but does not set variables or print errors. +dnl +dnl Please remember that m4 expands AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +dnl only at the first occurence in configure.ac, so if the first place +dnl it's called might be skipped (such as if it is within an "if", you +dnl have to call PKG_CHECK_EXISTS manually +AC_DEFUN([PKG_CHECK_EXISTS], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +if test -n "$PKG_CONFIG" && \ + AC_RUN_LOG([$PKG_CONFIG --exists --print-errors "$1"]); then + m4_default([$2], [:]) +m4_ifvaln([$3], [else + $3])dnl +fi]) + +dnl _PKG_CONFIG([VARIABLE], [COMMAND], [MODULES]) +dnl --------------------------------------------- +dnl Internal wrapper calling pkg-config via PKG_CONFIG and setting +dnl pkg_failed based on the result. +m4_define([_PKG_CONFIG], +[if test -n "$$1"; then + pkg_cv_[]$1="$$1" + elif test -n "$PKG_CONFIG"; then + PKG_CHECK_EXISTS([$3], + [pkg_cv_[]$1=`$PKG_CONFIG --[]$2 "$3" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes ], + [pkg_failed=yes]) + else + pkg_failed=untried +fi[]dnl +])dnl _PKG_CONFIG + +dnl _PKG_SHORT_ERRORS_SUPPORTED +dnl --------------------------- +dnl Internal check to see if pkg-config supports short errors. +AC_DEFUN([_PKG_SHORT_ERRORS_SUPPORTED], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG]) +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi[]dnl +])dnl _PKG_SHORT_ERRORS_SUPPORTED + + +dnl PKG_CHECK_MODULES(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl -------------------------------------------------------------- +dnl Since: 0.4.0 +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES might not happen, you should be sure to include an +dnl explicit call to PKG_PROG_PKG_CONFIG in your configure.ac +AC_DEFUN([PKG_CHECK_MODULES], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl +AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl + +pkg_failed=no +AC_MSG_CHECKING([for $1]) + +_PKG_CONFIG([$1][_CFLAGS], [cflags], [$2]) +_PKG_CONFIG([$1][_LIBS], [libs], [$2]) + +m4_define([_PKG_TEXT], [Alternatively, you may set the environment variables $1[]_CFLAGS +and $1[]_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details.]) + +if test $pkg_failed = yes; then + AC_MSG_RESULT([no]) + _PKG_SHORT_ERRORS_SUPPORTED + if test $_pkg_short_errors_supported = yes; then + $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1` + else + $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$$1[]_PKG_ERRORS" >&AS_MESSAGE_LOG_FD + + m4_default([$4], [AC_MSG_ERROR( +[Package requirements ($2) were not met: + +$$1_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +_PKG_TEXT])[]dnl + ]) +elif test $pkg_failed = untried; then + AC_MSG_RESULT([no]) + m4_default([$4], [AC_MSG_FAILURE( +[The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +_PKG_TEXT + +To get pkg-config, see .])[]dnl + ]) +else + $1[]_CFLAGS=$pkg_cv_[]$1[]_CFLAGS + $1[]_LIBS=$pkg_cv_[]$1[]_LIBS + AC_MSG_RESULT([yes]) + $3 +fi[]dnl +])dnl PKG_CHECK_MODULES + + +dnl PKG_CHECK_MODULES_STATIC(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], +dnl [ACTION-IF-NOT-FOUND]) +dnl --------------------------------------------------------------------- +dnl Since: 0.29 +dnl +dnl Checks for existence of MODULES and gathers its build flags with +dnl static libraries enabled. Sets VARIABLE-PREFIX_CFLAGS from --cflags +dnl and VARIABLE-PREFIX_LIBS from --libs. +dnl +dnl Note that if there is a possibility the first call to +dnl PKG_CHECK_MODULES_STATIC might not happen, you should be sure to +dnl include an explicit call to PKG_PROG_PKG_CONFIG in your +dnl configure.ac. +AC_DEFUN([PKG_CHECK_MODULES_STATIC], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +_save_PKG_CONFIG=$PKG_CONFIG +PKG_CONFIG="$PKG_CONFIG --static" +PKG_CHECK_MODULES($@) +PKG_CONFIG=$_save_PKG_CONFIG[]dnl +])dnl PKG_CHECK_MODULES_STATIC + + +dnl PKG_INSTALLDIR([DIRECTORY]) +dnl ------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable pkgconfigdir as the location where a module +dnl should install pkg-config .pc files. By default the directory is +dnl $libdir/pkgconfig, but the default can be changed by passing +dnl DIRECTORY. The user can override through the --with-pkgconfigdir +dnl parameter. +AC_DEFUN([PKG_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${libdir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([pkgconfigdir], + [AS_HELP_STRING([--with-pkgconfigdir], pkg_description)],, + [with_pkgconfigdir=]pkg_default) +AC_SUBST([pkgconfigdir], [$with_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_INSTALLDIR + + +dnl PKG_NOARCH_INSTALLDIR([DIRECTORY]) +dnl -------------------------------- +dnl Since: 0.27 +dnl +dnl Substitutes the variable noarch_pkgconfigdir as the location where a +dnl module should install arch-independent pkg-config .pc files. By +dnl default the directory is $datadir/pkgconfig, but the default can be +dnl changed by passing DIRECTORY. The user can override through the +dnl --with-noarch-pkgconfigdir parameter. +AC_DEFUN([PKG_NOARCH_INSTALLDIR], +[m4_pushdef([pkg_default], [m4_default([$1], ['${datadir}/pkgconfig'])]) +m4_pushdef([pkg_description], + [pkg-config arch-independent installation directory @<:@]pkg_default[@:>@]) +AC_ARG_WITH([noarch-pkgconfigdir], + [AS_HELP_STRING([--with-noarch-pkgconfigdir], pkg_description)],, + [with_noarch_pkgconfigdir=]pkg_default) +AC_SUBST([noarch_pkgconfigdir], [$with_noarch_pkgconfigdir]) +m4_popdef([pkg_default]) +m4_popdef([pkg_description]) +])dnl PKG_NOARCH_INSTALLDIR + + +dnl PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE, +dnl [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) +dnl ------------------------------------------- +dnl Since: 0.28 +dnl +dnl Retrieves the value of the pkg-config variable for the given module. +AC_DEFUN([PKG_CHECK_VAR], +[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl +AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl + +_PKG_CONFIG([$1], [variable="][$3]["], [$2]) +AS_VAR_COPY([$1], [pkg_cv_][$1]) + +AS_VAR_IF([$1], [""], [$5], [$4])dnl +])dnl PKG_CHECK_VAR + +dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND], +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------ +dnl +dnl Prepare a "--with-" configure option using the lowercase +dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and +dnl PKG_CHECK_MODULES in a single macro. +AC_DEFUN([PKG_WITH_MODULES], +[ +m4_pushdef([with_arg], m4_tolower([$1])) + +m4_pushdef([description], + [m4_default([$5], [build with ]with_arg[ support])]) + +m4_pushdef([def_arg], [m4_default([$6], [auto])]) +m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes]) +m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no]) + +m4_case(def_arg, + [yes],[m4_pushdef([with_without], [--without-]with_arg)], + [m4_pushdef([with_without],[--with-]with_arg)]) + +AC_ARG_WITH(with_arg, + AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),, + [AS_TR_SH([with_]with_arg)=def_arg]) + +AS_CASE([$AS_TR_SH([with_]with_arg)], + [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)], + [auto],[PKG_CHECK_MODULES([$1],[$2], + [m4_n([def_action_if_found]) $3], + [m4_n([def_action_if_not_found]) $4])]) + +m4_popdef([with_arg]) +m4_popdef([description]) +m4_popdef([def_arg]) + +])dnl PKG_WITH_MODULES + +dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ----------------------------------------------- +dnl +dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES +dnl check._[VARIABLE-PREFIX] is exported as make variable. +AC_DEFUN([PKG_HAVE_WITH_MODULES], +[ +PKG_WITH_MODULES([$1],[$2],,,[$3],[$4]) + +AM_CONDITIONAL([HAVE_][$1], + [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"]) +])dnl PKG_HAVE_WITH_MODULES + +dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES, +dnl [DESCRIPTION], [DEFAULT]) +dnl ------------------------------------------------------ +dnl +dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after +dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make +dnl and preprocessor variable. +AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES], +[ +PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4]) + +AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"], + [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])]) +])dnl PKG_HAVE_DEFINE_WITH_MODULES + diff --git a/configure b/configure index 75e2e296f10b1..3b39641bab37c 100755 --- a/configure +++ b/configure @@ -630,7 +630,6 @@ OPENSSL_RPATH OPENSSL_LDFLAGS OPENSSL_LIBS OPENSSL_INCLUDES -PKG_CONFIG ENSUREPIP SRCDIRS THREADHEADERS @@ -662,6 +661,9 @@ DTRACE TCLTK_LIBS TCLTK_INCLUDES LIBFFI_INCLUDEDIR +PKG_CONFIG_LIBDIR +PKG_CONFIG_PATH +PKG_CONFIG TZPATH SHLIBS CFLAGSFORSHARED @@ -873,7 +875,10 @@ LDFLAGS LIBS CPPFLAGS CPP -PROFILE_TASK' +PROFILE_TASK +PKG_CONFIG +PKG_CONFIG_PATH +PKG_CONFIG_LIBDIR' # Initialize some variables set by options. @@ -1638,6 +1643,11 @@ Some influential environment variables: CPP C preprocessor PROFILE_TASK Python args for PGO generation task + PKG_CONFIG path to pkg-config utility + PKG_CONFIG_PATH + directories to add to pkg-config's search path + PKG_CONFIG_LIBDIR + path overriding pkg-config's built-in search path Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. @@ -10621,7 +10631,126 @@ $as_echo "no" >&6; } fi -PKG_PROG_PKG_CONFIG + + + + + + + +if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then + if test -n "$ac_tool_prefix"; then + # Extract the first word of "${ac_tool_prefix}pkg-config", so it can be a program name with args. +set dummy ${ac_tool_prefix}pkg-config; ac_word=$2 +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 +$as_echo_n "checking for $ac_word... " >&6; } +if ${ac_cv_path_PKG_CONFIG+:} false; then : + $as_echo_n "(cached) " >&6 +else + case $PKG_CONFIG in + [\\/]* | ?:[\\/]*) + ac_cv_path_PKG_CONFIG="$PKG_CONFIG" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" + $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done + done +IFS=$as_save_IFS + + ;; +esac +fi +PKG_CONFIG=$ac_cv_path_PKG_CONFIG +if test -n "$PKG_CONFIG"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PKG_CONFIG" >&5 +$as_echo "$PKG_CONFIG" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi + + +fi +if test -z "$ac_cv_path_PKG_CONFIG"; then + ac_pt_PKG_CONFIG=$PKG_CONFIG + # Extract the first word of "pkg-config", so it can be a program name with args. +set dummy pkg-config; ac_word=$2 +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 +$as_echo_n "checking for $ac_word... " >&6; } +if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then : + $as_echo_n "(cached) " >&6 +else + case $ac_pt_PKG_CONFIG in + [\\/]* | ?:[\\/]*) + ac_cv_path_ac_pt_PKG_CONFIG="$ac_pt_PKG_CONFIG" # Let the user override the test with a path. + ;; + *) + as_save_IFS=$IFS; IFS=$PATH_SEPARATOR +for as_dir in $PATH +do + IFS=$as_save_IFS + test -z "$as_dir" && as_dir=. + for ac_exec_ext in '' $ac_executable_extensions; do + if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then + ac_cv_path_ac_pt_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" + $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 + break 2 + fi +done + done +IFS=$as_save_IFS + + ;; +esac +fi +ac_pt_PKG_CONFIG=$ac_cv_path_ac_pt_PKG_CONFIG +if test -n "$ac_pt_PKG_CONFIG"; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_pt_PKG_CONFIG" >&5 +$as_echo "$ac_pt_PKG_CONFIG" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi + + if test "x$ac_pt_PKG_CONFIG" = x; then + PKG_CONFIG="" + else + case $cross_compiling:$ac_tool_warned in +yes:) +{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 +$as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} +ac_tool_warned=yes ;; +esac + PKG_CONFIG=$ac_pt_PKG_CONFIG + fi +else + PKG_CONFIG="$ac_cv_path_PKG_CONFIG" +fi + +fi +if test -n "$PKG_CONFIG"; then + _pkg_min_version=0.9.0 + { $as_echo "$as_me:${as_lineno-$LINENO}: checking pkg-config is at least version $_pkg_min_version" >&5 +$as_echo_n "checking pkg-config is at least version $_pkg_min_version... " >&6; } + if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + PKG_CONFIG="" + fi +fi # Check for use of the system expat library { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-system-expat" >&5 From webhook-mailer at python.org Mon Oct 4 02:50:03 2021 From: webhook-mailer at python.org (njsmith) Date: Mon, 04 Oct 2021 06:50:03 -0000 Subject: [Python-checkins] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) Message-ID: https://github.com/python/cpython/commit/e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5 commit: e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5 branch: main author: John Belmonte committer: njsmith date: 2021-10-03T23:49:55-07:00 summary: bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) * bpo-44594: fix (Async)ExitStack handling of __context__ Make enter_context(foo()) / enter_async_context(foo()) equivalent to `[async] with foo()` regarding __context__ when an exception is raised. Previously exceptions would be caught and re-raised with the wrong context when explicitly overriding __context__ with None. files: A Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst M Lib/contextlib.py M Lib/test/test_contextlib.py M Lib/test/test_contextlib_async.py diff --git a/Lib/contextlib.py b/Lib/contextlib.py index 1384d8903d17b..d90ca5d8ef988 100644 --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -553,10 +553,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception @@ -693,10 +693,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py index 43b8507771e25..7982d9d835a2b 100644 --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -799,6 +799,40 @@ def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + def test_exit_exception_explicit_none_context(self): + # Ensure ExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @contextmanager + def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @contextmanager + def my_cm_with_exit_stack(): + with self.exit_stack() as stack: + stack.enter_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + def test_exit_exception_non_suppressing(self): # http://bugs.python.org/issue19092 def raise_exc(exc): diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py index c738bf3c0bdfe..c16c7ecd19a25 100644 --- a/Lib/test/test_contextlib_async.py +++ b/Lib/test/test_contextlib_async.py @@ -646,6 +646,41 @@ async def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + @_async_test + async def test_async_exit_exception_explicit_none_context(self): + # Ensure AsyncExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @asynccontextmanager + async def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @asynccontextmanager + async def my_cm_with_exit_stack(): + async with self.exit_stack() as stack: + await stack.enter_async_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + async with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + @_async_test async def test_instance_bypass_async(self): class Example(object): pass diff --git a/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst new file mode 100644 index 0000000000000..a2bfd8ff5b51b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst @@ -0,0 +1,3 @@ +Fix an edge case of :class:`ExitStack` and :class:`AsyncExitStack` exception +chaining. They will now match ``with`` block behavior when ``__context__`` is +explicitly set to ``None`` when the exception is in flight. From webhook-mailer at python.org Mon Oct 4 07:09:57 2021 From: webhook-mailer at python.org (JulienPalard) Date: Mon, 04 Oct 2021 11:09:57 -0000 Subject: [Python-checkins] bpo-28206: Document signals Handlers, Sigmasks and Signals enums (GH-28628) Message-ID: https://github.com/python/cpython/commit/9be930f9b169fb3d92693670ae069df902709b83 commit: 9be930f9b169fb3d92693670ae069df902709b83 branch: main author: Bibo-Joshi <22366557+Bibo-Joshi at users.noreply.github.com> committer: JulienPalard date: 2021-10-04T13:09:40+02:00 summary: bpo-28206: Document signals Handlers, Sigmasks and Signals enums (GH-28628) Co-authored-by: desbma files: M Doc/library/signal.rst diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 84a569d03eb29..63821866a012b 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -68,10 +68,34 @@ Module contents signal (SIG*), handler (:const:`SIG_DFL`, :const:`SIG_IGN`) and sigmask (:const:`SIG_BLOCK`, :const:`SIG_UNBLOCK`, :const:`SIG_SETMASK`) related constants listed below were turned into - :class:`enums `. + :class:`enums ` (:class:`Signals`, :class:`Handlers` and :class:`Sigmasks` respectively). :func:`getsignal`, :func:`pthread_sigmask`, :func:`sigpending` and :func:`sigwait` functions return human-readable - :class:`enums `. + :class:`enums ` as :class:`Signals` objects. + + +The signal module defines three enums: + +.. class:: Signals + + :class:`enum.IntEnum` collection of SIG* constants and the CTRL_* constants. + + .. versionadded:: 3.5 + +.. class:: Handlers + + :class:`enum.IntEnum` collection the constants :const:`SIG_DFL` and :const:`SIG_IGN`. + + .. versionadded:: 3.5 + +.. class:: Sigmasks + + :class:`enum.IntEnum` collection the constants :const:`SIG_BLOCK`, :const:`SIG_UNBLOCK` and :const:`SIG_SETMASK`. + + Availability: Unix. See the man page :manpage:`sigprocmask(3)` and + :manpage:`pthread_sigmask(3)` for further information. + + .. versionadded:: 3.5 The variables defined in the :mod:`signal` module are: @@ -618,8 +642,8 @@ The :mod:`signal` module defines the following functions: .. _signal-example: -Example -------- +Examples +-------- Here is a minimal example program. It uses the :func:`alarm` function to limit the time spent waiting to open a file; this is useful if the file is for a @@ -631,7 +655,8 @@ be sent, and the handler raises an exception. :: import signal, os def handler(signum, frame): - print('Signal handler called with signal', signum) + signame = signal.Signals(signum).name + print(f'Signal handler called with signal {signame} ({signum})') raise OSError("Couldn't open device!") # Set the signal handler and a 5-second alarm From webhook-mailer at python.org Mon Oct 4 07:11:31 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 04 Oct 2021 11:11:31 -0000 Subject: [Python-checkins] bpo-45355: More use of sizeof(_Py_CODEUNIT) (GH-28720) Message-ID: https://github.com/python/cpython/commit/252b7bcb236dc261f3af1275bc90f9a303d9648f commit: 252b7bcb236dc261f3af1275bc90f9a303d9648f branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-04T14:11:26+03:00 summary: bpo-45355: More use of sizeof(_Py_CODEUNIT) (GH-28720) files: M Objects/codeobject.c M Objects/frameobject.c M Python/compile.c diff --git a/Objects/codeobject.c b/Objects/codeobject.c index ad8f13a781b94..8de5c4d9c8a9d 100644 --- a/Objects/codeobject.c +++ b/Objects/codeobject.c @@ -656,15 +656,13 @@ _PyCode_Addr2Offset(PyCodeObject* co, int addrq) if (co->co_columntable == Py_None || addrq < 0) { return -1; } - if (addrq % 2 == 1) { - --addrq; - } - if (addrq >= PyBytes_GET_SIZE(co->co_columntable)) { + addrq /= sizeof(_Py_CODEUNIT); + if (addrq*2 >= PyBytes_GET_SIZE(co->co_columntable)) { return -1; } unsigned char* bytes = (unsigned char*)PyBytes_AS_STRING(co->co_columntable); - return bytes[addrq] - 1; + return bytes[addrq*2] - 1; } int @@ -673,15 +671,13 @@ _PyCode_Addr2EndOffset(PyCodeObject* co, int addrq) if (co->co_columntable == Py_None || addrq < 0) { return -1; } - if (addrq % 2 == 0) { - ++addrq; - } - if (addrq >= PyBytes_GET_SIZE(co->co_columntable)) { + addrq /= sizeof(_Py_CODEUNIT); + if (addrq*2+1 >= PyBytes_GET_SIZE(co->co_columntable)) { return -1; } unsigned char* bytes = (unsigned char*)PyBytes_AS_STRING(co->co_columntable); - return bytes[addrq] - 1; + return bytes[addrq*2+1] - 1; } void diff --git a/Objects/frameobject.c b/Objects/frameobject.c index b743dc72eee79..e4c16de66211d 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -373,8 +373,8 @@ marklines(PyCodeObject *code, int len) } while (PyLineTable_NextAddressRange(&bounds)) { - assert(bounds.ar_start/2 < len); - linestarts[bounds.ar_start/2] = bounds.ar_line; + assert(bounds.ar_start/(int)sizeof(_Py_CODEUNIT) < len); + linestarts[bounds.ar_start/sizeof(_Py_CODEUNIT)] = bounds.ar_line; } return linestarts; } diff --git a/Python/compile.c b/Python/compile.c index fdc2ce61a8e03..694da29b771d0 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -7081,7 +7081,7 @@ assemble_line_range(struct assembler* a, int current, PyObject** table, int* prev, int* start, int* offset) { int ldelta, bdelta; - bdelta = (a->a_offset - *start) * 2; + bdelta = (a->a_offset - *start) * sizeof(_Py_CODEUNIT); if (bdelta == 0) { return 1; } From webhook-mailer at python.org Mon Oct 4 07:13:51 2021 From: webhook-mailer at python.org (pablogsal) Date: Mon, 04 Oct 2021 11:13:51 -0000 Subject: [Python-checkins] Fix compiler warning in ceval.c regarding signed comparison (GH-28716) Message-ID: https://github.com/python/cpython/commit/07cf10bafc8f6e1fcc82c10d97d3452325fc7c04 commit: 07cf10bafc8f6e1fcc82c10d97d3452325fc7c04 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-04T12:13:46+01:00 summary: Fix compiler warning in ceval.c regarding signed comparison (GH-28716) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index c951e563cd7a1..8f65bb3aec4bc 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -7015,7 +7015,7 @@ maybe_dtrace_line(InterpreterFrame *frame, if (line != -1) { /* Trace backward edges or first instruction of a new line */ if (frame->f_lasti < instr_prev || - (line != lastline && frame->f_lasti*sizeof(_Py_CODEUNIT) == trace_info->bounds.ar_start)) + (line != lastline && frame->f_lasti*sizeof(_Py_CODEUNIT) == (unsigned int)trace_info->bounds.ar_start)) { co_filename = PyUnicode_AsUTF8(frame->f_code->co_filename); if (!co_filename) { From webhook-mailer at python.org Mon Oct 4 08:01:16 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 04 Oct 2021 12:01:16 -0000 Subject: [Python-checkins] [3.10] bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711). (GH-28718) Message-ID: https://github.com/python/cpython/commit/b5499784ec0aa24c8f0d91f2317cc53b7743ada9 commit: b5499784ec0aa24c8f0d91f2317cc53b7743ada9 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-04T15:01:11+03:00 summary: [3.10] bpo-45355: Use sizeof(_Py_CODEUNIT) instead of literal 2 for the size of the code unit (GH-28711). (GH-28718) (cherry picked from commit 60b9e040c9cf40e69f42c0008e564458aa0379e8) files: M Objects/frameobject.c M Python/ceval.c M Python/traceback.c diff --git a/Objects/frameobject.c b/Objects/frameobject.c index aa973016aefea..8974d37412356 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -46,7 +46,7 @@ PyFrame_GetLineNumber(PyFrameObject *f) return f->f_lineno; } else { - return PyCode_Addr2Line(f->f_code, f->f_lasti*2); + return PyCode_Addr2Line(f->f_code, f->f_lasti*sizeof(_Py_CODEUNIT)); } } @@ -68,7 +68,7 @@ frame_getlasti(PyFrameObject *f, void *closure) if (f->f_lasti < 0) { return PyLong_FromLong(-1); } - return PyLong_FromLong(f->f_lasti*2); + return PyLong_FromLong(f->f_lasti*sizeof(_Py_CODEUNIT)); } diff --git a/Python/ceval.c b/Python/ceval.c index 686250e1a8f94..624baf537518f 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5490,7 +5490,7 @@ call_trace(Py_tracefunc func, PyObject *obj, } else { initialize_trace_info(trace_info, frame); - frame->f_lineno = _PyCode_CheckLineNumber(frame->f_lasti*2, &trace_info->bounds); + frame->f_lineno = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &trace_info->bounds); } result = func(obj, frame, what, arg); frame->f_lineno = 0; @@ -5530,8 +5530,8 @@ maybe_call_line_trace(Py_tracefunc func, PyObject *obj, then call the trace function if we're tracing source lines. */ initialize_trace_info(trace_info, frame); - int lastline = _PyCode_CheckLineNumber(instr_prev*2, &trace_info->bounds); - int line = _PyCode_CheckLineNumber(frame->f_lasti*2, &trace_info->bounds); + int lastline = _PyCode_CheckLineNumber(instr_prev*sizeof(_Py_CODEUNIT), &trace_info->bounds); + int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &trace_info->bounds); if (line != -1 && frame->f_trace_lines) { /* Trace backward edges or if line number has changed */ if (frame->f_lasti < instr_prev || line != lastline) { @@ -6494,7 +6494,7 @@ maybe_dtrace_line(PyFrameObject *frame, instruction window, reset the window. */ initialize_trace_info(trace_info, frame); - int line = _PyCode_CheckLineNumber(frame->f_lasti*2, &trace_info->bounds); + int line = _PyCode_CheckLineNumber(frame->f_lasti*sizeof(_Py_CODEUNIT), &trace_info->bounds); /* If the last instruction falls at the start of a line or if it represents a jump backwards, update the frame's line number and call the trace function. */ diff --git a/Python/traceback.c b/Python/traceback.c index 9b23f45ba5bb8..284c18119daf0 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -234,7 +234,7 @@ _PyTraceBack_FromFrame(PyObject *tb_next, PyFrameObject *frame) assert(tb_next == NULL || PyTraceBack_Check(tb_next)); assert(frame != NULL); - return tb_create_raw((PyTracebackObject *)tb_next, frame, frame->f_lasti*2, + return tb_create_raw((PyTracebackObject *)tb_next, frame, frame->f_lasti*sizeof(_Py_CODEUNIT), PyFrame_GetLineNumber(frame)); } From webhook-mailer at python.org Mon Oct 4 10:07:29 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 04 Oct 2021 14:07:29 -0000 Subject: [Python-checkins] [3.10] bpo-45355: More use of sizeof(_Py_CODEUNIT) (GH-28720). (GH-28721) Message-ID: https://github.com/python/cpython/commit/1670d590fa6b817e0d3f091ea12aee9ae744875a commit: 1670d590fa6b817e0d3f091ea12aee9ae744875a branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-04T17:07:21+03:00 summary: [3.10] bpo-45355: More use of sizeof(_Py_CODEUNIT) (GH-28720). (GH-28721) (cherry picked from commit 252b7bcb236dc261f3af1275bc90f9a303d9648f) files: M Objects/frameobject.c M Python/compile.c diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 8974d37412356..d02cf9d3ba9f0 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -275,8 +275,8 @@ marklines(PyCodeObject *code, int len) } while (PyLineTable_NextAddressRange(&bounds)) { - assert(bounds.ar_start/2 < len); - linestarts[bounds.ar_start/2] = bounds.ar_line; + assert(bounds.ar_start/(int)sizeof(_Py_CODEUNIT) < len); + linestarts[bounds.ar_start/sizeof(_Py_CODEUNIT)] = bounds.ar_line; } return linestarts; } diff --git a/Python/compile.c b/Python/compile.c index b007859bd2e73..97aa2247f6095 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -6619,7 +6619,7 @@ static int assemble_line_range(struct assembler *a) { int ldelta, bdelta; - bdelta = (a->a_offset - a->a_lineno_start) * 2; + bdelta = (a->a_offset - a->a_lineno_start) * sizeof(_Py_CODEUNIT); if (bdelta == 0) { return 1; } From webhook-mailer at python.org Mon Oct 4 15:18:41 2021 From: webhook-mailer at python.org (pablogsal) Date: Mon, 04 Oct 2021 19:18:41 -0000 Subject: [Python-checkins] Fix minor typo in 3.10.rst (GH-28253) (GH-28259) Message-ID: https://github.com/python/cpython/commit/73af5ba83109118d6fb3367c59cacb4263143259 commit: 73af5ba83109118d6fb3367c59cacb4263143259 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: pablogsal date: 2021-09-14T19:58:05+01:00 summary: Fix minor typo in 3.10.rst (GH-28253) (GH-28259) (cherry picked from commit 73668541357caa813e7daa8792fab6fdf755a07f) Co-authored-by: D.Lintin Co-authored-by: D.Lintin files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index dbf89239d652e..5a5f4a360fb9a 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1843,7 +1843,7 @@ Changes in the Python syntax * Deprecation warning is now emitted when compiling previously valid syntax if the numeric literal is immediately followed by a keyword (like in ``0in x``). - If future releases it will be changed to syntax warning, and finally to a + In future releases it will be changed to syntax warning, and finally to a syntax error. To get rid of the warning and make the code compatible with future releases just add a space between the numeric literal and the following keyword. From webhook-mailer at python.org Mon Oct 4 15:27:38 2021 From: webhook-mailer at python.org (pablogsal) Date: Mon, 04 Oct 2021 19:27:38 -0000 Subject: [Python-checkins] Restore configure changes for pkg-config Message-ID: https://github.com/python/cpython/commit/47016dcdf82eff3dac6335cc5a6395824e466317 commit: 47016dcdf82eff3dac6335cc5a6395824e466317 branch: 3.10 author: Pablo Galindo committer: pablogsal date: 2021-10-04T20:27:22+01:00 summary: Restore configure changes for pkg-config files: M aclocal.m4 M configure diff --git a/aclocal.m4 b/aclocal.m4 index 2f1bd37528c85..987bfdf215ccb 100644 --- a/aclocal.m4 +++ b/aclocal.m4 @@ -275,347 +275,3 @@ AC_DEFUN([AX_CHECK_OPENSSL], [ AC_SUBST([OPENSSL_LDFLAGS]) ]) -# pkg.m4 - Macros to locate and utilise pkg-config. -*- Autoconf -*- -# serial 11 (pkg-config-0.29.1) - -dnl Copyright ? 2004 Scott James Remnant . -dnl Copyright ? 2012-2015 Dan Nicholson -dnl -dnl This program is free software; you can redistribute it and/or modify -dnl it under the terms of the GNU General Public License as published by -dnl the Free Software Foundation; either version 2 of the License, or -dnl (at your option) any later version. -dnl -dnl This program is distributed in the hope that it will be useful, but -dnl WITHOUT ANY WARRANTY; without even the implied warranty of -dnl MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -dnl General Public License for more details. -dnl -dnl You should have received a copy of the GNU General Public License -dnl along with this program; if not, write to the Free Software -dnl Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA -dnl 02111-1307, USA. -dnl -dnl As a special exception to the GNU General Public License, if you -dnl distribute this file as part of a program that contains a -dnl configuration script generated by Autoconf, you may include it under -dnl the same distribution terms that you use for the rest of that -dnl program. - -dnl PKG_PREREQ(MIN-VERSION) -dnl ----------------------- -dnl Since: 0.29 -dnl -dnl Verify that the version of the pkg-config macros are at least -dnl MIN-VERSION. Unlike PKG_PROG_PKG_CONFIG, which checks the user's -dnl installed version of pkg-config, this checks the developer's version -dnl of pkg.m4 when generating configure. -dnl -dnl To ensure that this macro is defined, also add: -dnl m4_ifndef([PKG_PREREQ], -dnl [m4_fatal([must install pkg-config 0.29 or later before running autoconf/autogen])]) -dnl -dnl See the "Since" comment for each macro you use to see what version -dnl of the macros you require. -m4_defun([PKG_PREREQ], -[m4_define([PKG_MACROS_VERSION], [0.29.1]) -m4_if(m4_version_compare(PKG_MACROS_VERSION, [$1]), -1, - [m4_fatal([pkg.m4 version $1 or higher is required but ]PKG_MACROS_VERSION[ found])]) -])dnl PKG_PREREQ - -dnl PKG_PROG_PKG_CONFIG([MIN-VERSION]) -dnl ---------------------------------- -dnl Since: 0.16 -dnl -dnl Search for the pkg-config tool and set the PKG_CONFIG variable to -dnl first found in the path. Checks that the version of pkg-config found -dnl is at least MIN-VERSION. If MIN-VERSION is not specified, 0.9.0 is -dnl used since that's the first version where most current features of -dnl pkg-config existed. -AC_DEFUN([PKG_PROG_PKG_CONFIG], -[m4_pattern_forbid([^_?PKG_[A-Z_]+$]) -m4_pattern_allow([^PKG_CONFIG(_(PATH|LIBDIR|SYSROOT_DIR|ALLOW_SYSTEM_(CFLAGS|LIBS)))?$]) -m4_pattern_allow([^PKG_CONFIG_(DISABLE_UNINSTALLED|TOP_BUILD_DIR|DEBUG_SPEW)$]) -AC_ARG_VAR([PKG_CONFIG], [path to pkg-config utility]) -AC_ARG_VAR([PKG_CONFIG_PATH], [directories to add to pkg-config's search path]) -AC_ARG_VAR([PKG_CONFIG_LIBDIR], [path overriding pkg-config's built-in search path]) - -if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then - AC_PATH_TOOL([PKG_CONFIG], [pkg-config]) -fi -if test -n "$PKG_CONFIG"; then - _pkg_min_version=m4_default([$1], [0.9.0]) - AC_MSG_CHECKING([pkg-config is at least version $_pkg_min_version]) - if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then - AC_MSG_RESULT([yes]) - else - AC_MSG_RESULT([no]) - PKG_CONFIG="" - fi -fi[]dnl -])dnl PKG_PROG_PKG_CONFIG - -dnl PKG_CHECK_EXISTS(MODULES, [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) -dnl ------------------------------------------------------------------- -dnl Since: 0.18 -dnl -dnl Check to see whether a particular set of modules exists. Similar to -dnl PKG_CHECK_MODULES(), but does not set variables or print errors. -dnl -dnl Please remember that m4 expands AC_REQUIRE([PKG_PROG_PKG_CONFIG]) -dnl only at the first occurence in configure.ac, so if the first place -dnl it's called might be skipped (such as if it is within an "if", you -dnl have to call PKG_CHECK_EXISTS manually -AC_DEFUN([PKG_CHECK_EXISTS], -[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl -if test -n "$PKG_CONFIG" && \ - AC_RUN_LOG([$PKG_CONFIG --exists --print-errors "$1"]); then - m4_default([$2], [:]) -m4_ifvaln([$3], [else - $3])dnl -fi]) - -dnl _PKG_CONFIG([VARIABLE], [COMMAND], [MODULES]) -dnl --------------------------------------------- -dnl Internal wrapper calling pkg-config via PKG_CONFIG and setting -dnl pkg_failed based on the result. -m4_define([_PKG_CONFIG], -[if test -n "$$1"; then - pkg_cv_[]$1="$$1" - elif test -n "$PKG_CONFIG"; then - PKG_CHECK_EXISTS([$3], - [pkg_cv_[]$1=`$PKG_CONFIG --[]$2 "$3" 2>/dev/null` - test "x$?" != "x0" && pkg_failed=yes ], - [pkg_failed=yes]) - else - pkg_failed=untried -fi[]dnl -])dnl _PKG_CONFIG - -dnl _PKG_SHORT_ERRORS_SUPPORTED -dnl --------------------------- -dnl Internal check to see if pkg-config supports short errors. -AC_DEFUN([_PKG_SHORT_ERRORS_SUPPORTED], -[AC_REQUIRE([PKG_PROG_PKG_CONFIG]) -if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then - _pkg_short_errors_supported=yes -else - _pkg_short_errors_supported=no -fi[]dnl -])dnl _PKG_SHORT_ERRORS_SUPPORTED - - -dnl PKG_CHECK_MODULES(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], -dnl [ACTION-IF-NOT-FOUND]) -dnl -------------------------------------------------------------- -dnl Since: 0.4.0 -dnl -dnl Note that if there is a possibility the first call to -dnl PKG_CHECK_MODULES might not happen, you should be sure to include an -dnl explicit call to PKG_PROG_PKG_CONFIG in your configure.ac -AC_DEFUN([PKG_CHECK_MODULES], -[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl -AC_ARG_VAR([$1][_CFLAGS], [C compiler flags for $1, overriding pkg-config])dnl -AC_ARG_VAR([$1][_LIBS], [linker flags for $1, overriding pkg-config])dnl - -pkg_failed=no -AC_MSG_CHECKING([for $1]) - -_PKG_CONFIG([$1][_CFLAGS], [cflags], [$2]) -_PKG_CONFIG([$1][_LIBS], [libs], [$2]) - -m4_define([_PKG_TEXT], [Alternatively, you may set the environment variables $1[]_CFLAGS -and $1[]_LIBS to avoid the need to call pkg-config. -See the pkg-config man page for more details.]) - -if test $pkg_failed = yes; then - AC_MSG_RESULT([no]) - _PKG_SHORT_ERRORS_SUPPORTED - if test $_pkg_short_errors_supported = yes; then - $1[]_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "$2" 2>&1` - else - $1[]_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "$2" 2>&1` - fi - # Put the nasty error message in config.log where it belongs - echo "$$1[]_PKG_ERRORS" >&AS_MESSAGE_LOG_FD - - m4_default([$4], [AC_MSG_ERROR( -[Package requirements ($2) were not met: - -$$1_PKG_ERRORS - -Consider adjusting the PKG_CONFIG_PATH environment variable if you -installed software in a non-standard prefix. - -_PKG_TEXT])[]dnl - ]) -elif test $pkg_failed = untried; then - AC_MSG_RESULT([no]) - m4_default([$4], [AC_MSG_FAILURE( -[The pkg-config script could not be found or is too old. Make sure it -is in your PATH or set the PKG_CONFIG environment variable to the full -path to pkg-config. - -_PKG_TEXT - -To get pkg-config, see .])[]dnl - ]) -else - $1[]_CFLAGS=$pkg_cv_[]$1[]_CFLAGS - $1[]_LIBS=$pkg_cv_[]$1[]_LIBS - AC_MSG_RESULT([yes]) - $3 -fi[]dnl -])dnl PKG_CHECK_MODULES - - -dnl PKG_CHECK_MODULES_STATIC(VARIABLE-PREFIX, MODULES, [ACTION-IF-FOUND], -dnl [ACTION-IF-NOT-FOUND]) -dnl --------------------------------------------------------------------- -dnl Since: 0.29 -dnl -dnl Checks for existence of MODULES and gathers its build flags with -dnl static libraries enabled. Sets VARIABLE-PREFIX_CFLAGS from --cflags -dnl and VARIABLE-PREFIX_LIBS from --libs. -dnl -dnl Note that if there is a possibility the first call to -dnl PKG_CHECK_MODULES_STATIC might not happen, you should be sure to -dnl include an explicit call to PKG_PROG_PKG_CONFIG in your -dnl configure.ac. -AC_DEFUN([PKG_CHECK_MODULES_STATIC], -[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl -_save_PKG_CONFIG=$PKG_CONFIG -PKG_CONFIG="$PKG_CONFIG --static" -PKG_CHECK_MODULES($@) -PKG_CONFIG=$_save_PKG_CONFIG[]dnl -])dnl PKG_CHECK_MODULES_STATIC - - -dnl PKG_INSTALLDIR([DIRECTORY]) -dnl ------------------------- -dnl Since: 0.27 -dnl -dnl Substitutes the variable pkgconfigdir as the location where a module -dnl should install pkg-config .pc files. By default the directory is -dnl $libdir/pkgconfig, but the default can be changed by passing -dnl DIRECTORY. The user can override through the --with-pkgconfigdir -dnl parameter. -AC_DEFUN([PKG_INSTALLDIR], -[m4_pushdef([pkg_default], [m4_default([$1], ['${libdir}/pkgconfig'])]) -m4_pushdef([pkg_description], - [pkg-config installation directory @<:@]pkg_default[@:>@]) -AC_ARG_WITH([pkgconfigdir], - [AS_HELP_STRING([--with-pkgconfigdir], pkg_description)],, - [with_pkgconfigdir=]pkg_default) -AC_SUBST([pkgconfigdir], [$with_pkgconfigdir]) -m4_popdef([pkg_default]) -m4_popdef([pkg_description]) -])dnl PKG_INSTALLDIR - - -dnl PKG_NOARCH_INSTALLDIR([DIRECTORY]) -dnl -------------------------------- -dnl Since: 0.27 -dnl -dnl Substitutes the variable noarch_pkgconfigdir as the location where a -dnl module should install arch-independent pkg-config .pc files. By -dnl default the directory is $datadir/pkgconfig, but the default can be -dnl changed by passing DIRECTORY. The user can override through the -dnl --with-noarch-pkgconfigdir parameter. -AC_DEFUN([PKG_NOARCH_INSTALLDIR], -[m4_pushdef([pkg_default], [m4_default([$1], ['${datadir}/pkgconfig'])]) -m4_pushdef([pkg_description], - [pkg-config arch-independent installation directory @<:@]pkg_default[@:>@]) -AC_ARG_WITH([noarch-pkgconfigdir], - [AS_HELP_STRING([--with-noarch-pkgconfigdir], pkg_description)],, - [with_noarch_pkgconfigdir=]pkg_default) -AC_SUBST([noarch_pkgconfigdir], [$with_noarch_pkgconfigdir]) -m4_popdef([pkg_default]) -m4_popdef([pkg_description]) -])dnl PKG_NOARCH_INSTALLDIR - - -dnl PKG_CHECK_VAR(VARIABLE, MODULE, CONFIG-VARIABLE, -dnl [ACTION-IF-FOUND], [ACTION-IF-NOT-FOUND]) -dnl ------------------------------------------- -dnl Since: 0.28 -dnl -dnl Retrieves the value of the pkg-config variable for the given module. -AC_DEFUN([PKG_CHECK_VAR], -[AC_REQUIRE([PKG_PROG_PKG_CONFIG])dnl -AC_ARG_VAR([$1], [value of $3 for $2, overriding pkg-config])dnl - -_PKG_CONFIG([$1], [variable="][$3]["], [$2]) -AS_VAR_COPY([$1], [pkg_cv_][$1]) - -AS_VAR_IF([$1], [""], [$5], [$4])dnl -])dnl PKG_CHECK_VAR - -dnl PKG_WITH_MODULES(VARIABLE-PREFIX, MODULES, -dnl [ACTION-IF-FOUND],[ACTION-IF-NOT-FOUND], -dnl [DESCRIPTION], [DEFAULT]) -dnl ------------------------------------------ -dnl -dnl Prepare a "--with-" configure option using the lowercase -dnl [VARIABLE-PREFIX] name, merging the behaviour of AC_ARG_WITH and -dnl PKG_CHECK_MODULES in a single macro. -AC_DEFUN([PKG_WITH_MODULES], -[ -m4_pushdef([with_arg], m4_tolower([$1])) - -m4_pushdef([description], - [m4_default([$5], [build with ]with_arg[ support])]) - -m4_pushdef([def_arg], [m4_default([$6], [auto])]) -m4_pushdef([def_action_if_found], [AS_TR_SH([with_]with_arg)=yes]) -m4_pushdef([def_action_if_not_found], [AS_TR_SH([with_]with_arg)=no]) - -m4_case(def_arg, - [yes],[m4_pushdef([with_without], [--without-]with_arg)], - [m4_pushdef([with_without],[--with-]with_arg)]) - -AC_ARG_WITH(with_arg, - AS_HELP_STRING(with_without, description[ @<:@default=]def_arg[@:>@]),, - [AS_TR_SH([with_]with_arg)=def_arg]) - -AS_CASE([$AS_TR_SH([with_]with_arg)], - [yes],[PKG_CHECK_MODULES([$1],[$2],$3,$4)], - [auto],[PKG_CHECK_MODULES([$1],[$2], - [m4_n([def_action_if_found]) $3], - [m4_n([def_action_if_not_found]) $4])]) - -m4_popdef([with_arg]) -m4_popdef([description]) -m4_popdef([def_arg]) - -])dnl PKG_WITH_MODULES - -dnl PKG_HAVE_WITH_MODULES(VARIABLE-PREFIX, MODULES, -dnl [DESCRIPTION], [DEFAULT]) -dnl ----------------------------------------------- -dnl -dnl Convenience macro to trigger AM_CONDITIONAL after PKG_WITH_MODULES -dnl check._[VARIABLE-PREFIX] is exported as make variable. -AC_DEFUN([PKG_HAVE_WITH_MODULES], -[ -PKG_WITH_MODULES([$1],[$2],,,[$3],[$4]) - -AM_CONDITIONAL([HAVE_][$1], - [test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"]) -])dnl PKG_HAVE_WITH_MODULES - -dnl PKG_HAVE_DEFINE_WITH_MODULES(VARIABLE-PREFIX, MODULES, -dnl [DESCRIPTION], [DEFAULT]) -dnl ------------------------------------------------------ -dnl -dnl Convenience macro to run AM_CONDITIONAL and AC_DEFINE after -dnl PKG_WITH_MODULES check. HAVE_[VARIABLE-PREFIX] is exported as make -dnl and preprocessor variable. -AC_DEFUN([PKG_HAVE_DEFINE_WITH_MODULES], -[ -PKG_HAVE_WITH_MODULES([$1],[$2],[$3],[$4]) - -AS_IF([test "$AS_TR_SH([with_]m4_tolower([$1]))" = "yes"], - [AC_DEFINE([HAVE_][$1], 1, [Enable ]m4_tolower([$1])[ support])]) -])dnl PKG_HAVE_DEFINE_WITH_MODULES - diff --git a/configure b/configure index 02d882ed39d68..753f9564693bc 100755 --- a/configure +++ b/configure @@ -630,6 +630,7 @@ OPENSSL_RPATH OPENSSL_LDFLAGS OPENSSL_LIBS OPENSSL_INCLUDES +PKG_CONFIG ENSUREPIP SRCDIRS THREADHEADERS @@ -661,9 +662,6 @@ DTRACE TCLTK_LIBS TCLTK_INCLUDES LIBFFI_INCLUDEDIR -PKG_CONFIG_LIBDIR -PKG_CONFIG_PATH -PKG_CONFIG TZPATH SHLIBS CFLAGSFORSHARED @@ -875,10 +873,7 @@ LDFLAGS LIBS CPPFLAGS CPP -PROFILE_TASK -PKG_CONFIG -PKG_CONFIG_PATH -PKG_CONFIG_LIBDIR' +PROFILE_TASK' # Initialize some variables set by options. @@ -1642,11 +1637,6 @@ Some influential environment variables: CPP C preprocessor PROFILE_TASK Python args for PGO generation task - PKG_CONFIG path to pkg-config utility - PKG_CONFIG_PATH - directories to add to pkg-config's search path - PKG_CONFIG_LIBDIR - path overriding pkg-config's built-in search path Use these variables to override the choices made by `configure' or to help it to find libraries and programs with nonstandard names/locations. @@ -10552,126 +10542,7 @@ $as_echo "no" >&6; } fi - - - - - - - -if test "x$ac_cv_env_PKG_CONFIG_set" != "xset"; then - if test -n "$ac_tool_prefix"; then - # Extract the first word of "${ac_tool_prefix}pkg-config", so it can be a program name with args. -set dummy ${ac_tool_prefix}pkg-config; ac_word=$2 -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 -$as_echo_n "checking for $ac_word... " >&6; } -if ${ac_cv_path_PKG_CONFIG+:} false; then : - $as_echo_n "(cached) " >&6 -else - case $PKG_CONFIG in - [\\/]* | ?:[\\/]*) - ac_cv_path_PKG_CONFIG="$PKG_CONFIG" # Let the user override the test with a path. - ;; - *) - as_save_IFS=$IFS; IFS=$PATH_SEPARATOR -for as_dir in $PATH -do - IFS=$as_save_IFS - test -z "$as_dir" && as_dir=. - for ac_exec_ext in '' $ac_executable_extensions; do - if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then - ac_cv_path_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" - $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 - break 2 - fi -done - done -IFS=$as_save_IFS - - ;; -esac -fi -PKG_CONFIG=$ac_cv_path_PKG_CONFIG -if test -n "$PKG_CONFIG"; then - { $as_echo "$as_me:${as_lineno-$LINENO}: result: $PKG_CONFIG" >&5 -$as_echo "$PKG_CONFIG" >&6; } -else - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } -fi - - -fi -if test -z "$ac_cv_path_PKG_CONFIG"; then - ac_pt_PKG_CONFIG=$PKG_CONFIG - # Extract the first word of "pkg-config", so it can be a program name with args. -set dummy pkg-config; ac_word=$2 -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for $ac_word" >&5 -$as_echo_n "checking for $ac_word... " >&6; } -if ${ac_cv_path_ac_pt_PKG_CONFIG+:} false; then : - $as_echo_n "(cached) " >&6 -else - case $ac_pt_PKG_CONFIG in - [\\/]* | ?:[\\/]*) - ac_cv_path_ac_pt_PKG_CONFIG="$ac_pt_PKG_CONFIG" # Let the user override the test with a path. - ;; - *) - as_save_IFS=$IFS; IFS=$PATH_SEPARATOR -for as_dir in $PATH -do - IFS=$as_save_IFS - test -z "$as_dir" && as_dir=. - for ac_exec_ext in '' $ac_executable_extensions; do - if as_fn_executable_p "$as_dir/$ac_word$ac_exec_ext"; then - ac_cv_path_ac_pt_PKG_CONFIG="$as_dir/$ac_word$ac_exec_ext" - $as_echo "$as_me:${as_lineno-$LINENO}: found $as_dir/$ac_word$ac_exec_ext" >&5 - break 2 - fi -done - done -IFS=$as_save_IFS - - ;; -esac -fi -ac_pt_PKG_CONFIG=$ac_cv_path_ac_pt_PKG_CONFIG -if test -n "$ac_pt_PKG_CONFIG"; then - { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_pt_PKG_CONFIG" >&5 -$as_echo "$ac_pt_PKG_CONFIG" >&6; } -else - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } -fi - - if test "x$ac_pt_PKG_CONFIG" = x; then - PKG_CONFIG="" - else - case $cross_compiling:$ac_tool_warned in -yes:) -{ $as_echo "$as_me:${as_lineno-$LINENO}: WARNING: using cross tools not prefixed with host triplet" >&5 -$as_echo "$as_me: WARNING: using cross tools not prefixed with host triplet" >&2;} -ac_tool_warned=yes ;; -esac - PKG_CONFIG=$ac_pt_PKG_CONFIG - fi -else - PKG_CONFIG="$ac_cv_path_PKG_CONFIG" -fi - -fi -if test -n "$PKG_CONFIG"; then - _pkg_min_version=0.9.0 - { $as_echo "$as_me:${as_lineno-$LINENO}: checking pkg-config is at least version $_pkg_min_version" >&5 -$as_echo_n "checking pkg-config is at least version $_pkg_min_version... " >&6; } - if $PKG_CONFIG --atleast-pkgconfig-version $_pkg_min_version; then - { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 -$as_echo "yes" >&6; } - else - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } - PKG_CONFIG="" - fi -fi +PKG_PROG_PKG_CONFIG # Check for use of the system expat library { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-system-expat" >&5 From webhook-mailer at python.org Tue Oct 5 02:21:48 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 06:21:48 -0000 Subject: [Python-checkins] [3.10] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28730) Message-ID: https://github.com/python/cpython/commit/872b1e537e96d0dc4ff37c738031940b5d271366 commit: 872b1e537e96d0dc4ff37c738031940b5d271366 branch: 3.10 author: John Belmonte committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-04T23:21:34-07:00 summary: [3.10] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28730) Make enter_context(foo()) / enter_async_context(foo()) equivalent to `[async] with foo()` regarding __context__ when an exception is raised. Previously exceptions would be caught and re-raised with the wrong context when explicitly overriding __context__ with None.. (cherry picked from commit e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5) Co-authored-by: John Belmonte Automerge-Triggered-By: GH:njsmith files: A Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst M Lib/contextlib.py M Lib/test/test_contextlib.py M Lib/test/test_contextlib_async.py diff --git a/Lib/contextlib.py b/Lib/contextlib.py index 1e0a39f8d8512..c63a8492e2d3d 100644 --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -540,10 +540,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception @@ -674,10 +674,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py index 5a080654acb6c..fbaae2dd09fd9 100644 --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -780,6 +780,40 @@ def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + def test_exit_exception_explicit_none_context(self): + # Ensure ExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @contextmanager + def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @contextmanager + def my_cm_with_exit_stack(): + with self.exit_stack() as stack: + stack.enter_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + def test_exit_exception_non_suppressing(self): # http://bugs.python.org/issue19092 def raise_exc(exc): diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py index 603162eaeaa7b..127d7500656a5 100644 --- a/Lib/test/test_contextlib_async.py +++ b/Lib/test/test_contextlib_async.py @@ -552,6 +552,41 @@ async def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + @_async_test + async def test_async_exit_exception_explicit_none_context(self): + # Ensure AsyncExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @asynccontextmanager + async def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @asynccontextmanager + async def my_cm_with_exit_stack(): + async with self.exit_stack() as stack: + await stack.enter_async_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + async with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + class TestAsyncNullcontext(unittest.TestCase): @_async_test diff --git a/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst new file mode 100644 index 0000000000000..a2bfd8ff5b51b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst @@ -0,0 +1,3 @@ +Fix an edge case of :class:`ExitStack` and :class:`AsyncExitStack` exception +chaining. They will now match ``with`` block behavior when ``__context__`` is +explicitly set to ``None`` when the exception is in flight. From webhook-mailer at python.org Tue Oct 5 02:37:31 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 06:37:31 -0000 Subject: [Python-checkins] [3.9] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28731) Message-ID: https://github.com/python/cpython/commit/7c2a040a10654d67ff543a55858ba2d7a9f7eea8 commit: 7c2a040a10654d67ff543a55858ba2d7a9f7eea8 branch: 3.9 author: John Belmonte committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-04T23:37:24-07:00 summary: [3.9] bpo-44594: fix (Async)ExitStack handling of __context__ (gh-27089) (GH-28731) Make enter_context(foo()) / enter_async_context(foo()) equivalent to `[async] with foo()` regarding __context__ when an exception is raised. Previously exceptions would be caught and re-raised with the wrong context when explicitly overriding __context__ with None.. (cherry picked from commit e6d1aa1ac65b6908fdea2c70ec3aa8c4f1dffcb5) Co-authored-by: John Belmonte Automerge-Triggered-By: GH:njsmith files: A Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst M Lib/contextlib.py M Lib/test/test_contextlib.py M Lib/test/test_contextlib_async.py diff --git a/Lib/contextlib.py b/Lib/contextlib.py index f60f0c274a715..4e8f5f7593917 100644 --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -496,10 +496,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception @@ -630,10 +630,10 @@ def _fix_exception_context(new_exc, old_exc): # Context may not be correct, so find the end of the chain while 1: exc_context = new_exc.__context__ - if exc_context is old_exc: + if exc_context is None or exc_context is old_exc: # Context is already set correctly (see issue 20317) return - if exc_context is None or exc_context is frame_exc: + if exc_context is frame_exc: break new_exc = exc_context # Change the end of the chain to point to the exception diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py index ad071bb2a22d3..354ea8b3c32b7 100644 --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -777,6 +777,40 @@ def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + def test_exit_exception_explicit_none_context(self): + # Ensure ExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @contextmanager + def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @contextmanager + def my_cm_with_exit_stack(): + with self.exit_stack() as stack: + stack.enter_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + def test_exit_exception_non_suppressing(self): # http://bugs.python.org/issue19092 def raise_exc(exc): diff --git a/Lib/test/test_contextlib_async.py b/Lib/test/test_contextlib_async.py index 9d6854c702892..43aaf04744431 100644 --- a/Lib/test/test_contextlib_async.py +++ b/Lib/test/test_contextlib_async.py @@ -463,6 +463,41 @@ async def suppress_exc(*exc_details): self.assertIsInstance(inner_exc, ValueError) self.assertIsInstance(inner_exc.__context__, ZeroDivisionError) + @_async_test + async def test_async_exit_exception_explicit_none_context(self): + # Ensure AsyncExitStack chaining matches actual nested `with` statements + # regarding explicit __context__ = None. + + class MyException(Exception): + pass + + @asynccontextmanager + async def my_cm(): + try: + yield + except BaseException: + exc = MyException() + try: + raise exc + finally: + exc.__context__ = None + + @asynccontextmanager + async def my_cm_with_exit_stack(): + async with self.exit_stack() as stack: + await stack.enter_async_context(my_cm()) + yield stack + + for cm in (my_cm, my_cm_with_exit_stack): + with self.subTest(): + try: + async with cm(): + raise IndexError() + except MyException as exc: + self.assertIsNone(exc.__context__) + else: + self.fail("Expected IndexError, but no exception was raised") + if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst new file mode 100644 index 0000000000000..a2bfd8ff5b51b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst @@ -0,0 +1,3 @@ +Fix an edge case of :class:`ExitStack` and :class:`AsyncExitStack` exception +chaining. They will now match ``with`` block behavior when ``__context__`` is +explicitly set to ``None`` when the exception is in flight. From webhook-mailer at python.org Tue Oct 5 05:43:51 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 09:43:51 -0000 Subject: [Python-checkins] bpo-45371: Fix distutils' rpath support for clang (GH-28732) Message-ID: https://github.com/python/cpython/commit/ef6196028f966f22d82930b66e1371e75c5df2f7 commit: ef6196028f966f22d82930b66e1371e75c5df2f7 branch: main author: Christian Heimes committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-05T02:43:46-07:00 summary: bpo-45371: Fix distutils' rpath support for clang (GH-28732) Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst M Lib/distutils/unixccompiler.py diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py index f0792de74a1a4..d00c48981eb6d 100644 --- a/Lib/distutils/unixccompiler.py +++ b/Lib/distutils/unixccompiler.py @@ -215,7 +215,8 @@ def library_dir_option(self, dir): return "-L" + dir def _is_gcc(self, compiler_name): - return "gcc" in compiler_name or "g++" in compiler_name + # clang uses same syntax for rpath as gcc + return any(name in compiler_name for name in ("gcc", "g++", "clang")) def runtime_library_dir_option(self, dir): # XXX Hackish, at the very least. See Python bug #445902: diff --git a/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst new file mode 100644 index 0000000000000..045489be81a19 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst @@ -0,0 +1,3 @@ +Fix clang rpath issue in :mod:`distutils`. The UnixCCompiler now uses +correct clang option to add a runtime library directory (rpath) to a shared +library. From webhook-mailer at python.org Tue Oct 5 06:01:16 2021 From: webhook-mailer at python.org (markshannon) Date: Tue, 05 Oct 2021 10:01:16 -0000 Subject: [Python-checkins] bpo-43760: Check for tracing using 'bitwise or' instead of branch in dispatch. (GH-28723) Message-ID: https://github.com/python/cpython/commit/bd627eb7ed08a891dd1356756feb1ce2600358e4 commit: bd627eb7ed08a891dd1356756feb1ce2600358e4 branch: main author: Mark Shannon committer: markshannon date: 2021-10-05T11:01:11+01:00 summary: bpo-43760: Check for tracing using 'bitwise or' instead of branch in dispatch. (GH-28723) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst M Include/opcode.h M Python/ceval.c M Python/makeopcodetargets.py M Python/opcode_targets.h M Python/sysmodule.c M Tools/scripts/generate_opcode_h.py diff --git a/Include/opcode.h b/Include/opcode.h index 2789525594783..8817a4d650fd1 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -167,6 +167,7 @@ extern "C" { #define LOAD_FAST__LOAD_CONST 134 #define LOAD_CONST__LOAD_FAST 140 #define STORE_FAST__STORE_FAST 143 +#define DO_TRACING 255 #ifdef NEED_OPCODE_JUMP_TABLES static uint32_t _PyOpcode_RelativeJump[8] = { 0U, diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst new file mode 100644 index 0000000000000..1809b42b94438 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst @@ -0,0 +1,3 @@ +The number of hardware branches per instruction dispatch is reduced from two +to one by adding a special instruction for tracing. Patch by Mark Shannon. + diff --git a/Python/ceval.c b/Python/ceval.c index 8f65bb3aec4bc..2dbc291897c03 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1218,7 +1218,7 @@ eval_frame_handle_pending(PyThreadState *tstate) #endif #ifdef WITH_DTRACE -#define OR_DTRACE_LINE || PyDTrace_LINE_ENABLED() +#define OR_DTRACE_LINE | (PyDTrace_LINE_ENABLED() ? 255 : 0) #else #define OR_DTRACE_LINE #endif @@ -1240,11 +1240,13 @@ eval_frame_handle_pending(PyThreadState *tstate) #define USE_COMPUTED_GOTOS 0 #endif +#define INSTRUCTION_START() frame->f_lasti = INSTR_OFFSET(); next_instr++ + #if USE_COMPUTED_GOTOS -#define TARGET(op) TARGET_##op +#define TARGET(op) TARGET_##op: INSTRUCTION_START(); #define DISPATCH_GOTO() goto *opcode_targets[opcode] #else -#define TARGET(op) case op +#define TARGET(op) case op: INSTRUCTION_START(); #define DISPATCH_GOTO() goto dispatch_opcode #endif @@ -1274,12 +1276,11 @@ eval_frame_handle_pending(PyThreadState *tstate) #endif #endif #ifndef PRE_DISPATCH_GOTO -#define PRE_DISPATCH_GOTO() do { LLTRACE_INSTR(); RECORD_DXPROFILE(); } while (0) +#define PRE_DISPATCH_GOTO() do { LLTRACE_INSTR(); RECORD_DXPROFILE(); } while (0) #endif #define NOTRACE_DISPATCH() \ { \ - frame->f_lasti = INSTR_OFFSET(); \ NEXTOPARG(); \ PRE_DISPATCH_GOTO(); \ DISPATCH_GOTO(); \ @@ -1288,10 +1289,11 @@ eval_frame_handle_pending(PyThreadState *tstate) /* Do interpreter dispatch accounting for tracing and instrumentation */ #define DISPATCH() \ { \ - if (cframe.use_tracing OR_DTRACE_LINE) { \ - goto tracing_dispatch; \ - } \ - NOTRACE_DISPATCH(); \ + NEXTOPARG(); \ + PRE_DISPATCH_GOTO(); \ + assert(cframe.use_tracing == 0 || cframe.use_tracing == 255); \ + opcode |= cframe.use_tracing OR_DTRACE_LINE; \ + DISPATCH_GOTO(); \ } #define CHECK_EVAL_BREAKER() \ @@ -1316,7 +1318,6 @@ eval_frame_handle_pending(PyThreadState *tstate) _Py_CODEUNIT word = *next_instr; \ opcode = _Py_OPCODE(word); \ oparg = _Py_OPARG(word); \ - next_instr++; \ } while (0) #define JUMPTO(x) (next_instr = first_instr + (x)) #define JUMPBY(x) (next_instr += (x)) @@ -1326,7 +1327,6 @@ eval_frame_handle_pending(PyThreadState *tstate) _Py_CODEUNIT word = ((_Py_CODEUNIT *)PyBytes_AS_STRING(co->co_code))[INSTR_OFFSET()]; \ opcode = _Py_OPCODE(word); \ oparg = _Py_OPARG(word); \ - next_instr++; \ } while (0) /* OpCode prediction macros @@ -1363,10 +1363,10 @@ eval_frame_handle_pending(PyThreadState *tstate) #define PREDICT(op) \ do { \ _Py_CODEUNIT word = *next_instr; \ - opcode = _Py_OPCODE(word); \ + opcode = _Py_OPCODE(word) | cframe.use_tracing OR_DTRACE_LINE; \ if (opcode == op) { \ oparg = _Py_OPARG(word); \ - next_instr++; \ + INSTRUCTION_START(); \ goto PREDICT_ID(op); \ } \ } while(0) @@ -1516,6 +1516,15 @@ trace_function_entry(PyThreadState *tstate, InterpreterFrame *frame) return 0; } +static int +skip_backwards_over_extended_args(PyCodeObject *code, int offset) { + _Py_CODEUNIT *instrs = (_Py_CODEUNIT *)PyBytes_AS_STRING(code->co_code); + while (offset > 0 && _Py_OPCODE(instrs[offset-1]) == EXTENDED_ARG) { + offset--; + } + return offset; +} + PyObject* _Py_HOT_FUNCTION _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int throwflag) { @@ -1668,42 +1677,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); - tracing_dispatch: - { - int instr_prev = frame->f_lasti; - frame->f_lasti = INSTR_OFFSET(); - TRACING_NEXTOPARG(); - - if (PyDTrace_LINE_ENABLED()) - maybe_dtrace_line(frame, &tstate->trace_info, instr_prev); - - /* line-by-line tracing support */ - - if (cframe.use_tracing && - tstate->c_tracefunc != NULL && !tstate->tracing) { - int err; - /* see maybe_call_line_trace() - for expository comments */ - _PyFrame_SetStackPointer(frame, stack_pointer); - - err = maybe_call_line_trace(tstate->c_tracefunc, - tstate->c_traceobj, - tstate, frame, instr_prev); - if (err) { - /* trace function raised an exception */ - goto error; - } - /* Reload possibly changed frame fields */ - JUMPTO(frame->f_lasti); - - stack_pointer = _PyFrame_GetStackPointer(frame); - frame->stacktop = -1; - TRACING_NEXTOPARG(); - } - PRE_DISPATCH_GOTO(); - DISPATCH_GOTO(); - } - /* Start instructions */ #if USE_COMPUTED_GOTOS { @@ -1716,13 +1689,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr It is essential that any operation that fails must goto error and that all operation that succeed call DISPATCH() ! */ - TARGET(NOP): { + TARGET(NOP) { DISPATCH(); } /* We keep LOAD_CLOSURE so that the bytecode stays more readable. */ - TARGET(LOAD_CLOSURE): - TARGET(LOAD_FAST): { + TARGET(LOAD_CLOSURE) { PyObject *value = GETLOCAL(oparg); if (value == NULL) { goto unbound_local_error; @@ -1732,7 +1704,17 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_CONST): { + TARGET(LOAD_FAST) { + PyObject *value = GETLOCAL(oparg); + if (value == NULL) { + goto unbound_local_error; + } + Py_INCREF(value); + PUSH(value); + DISPATCH(); + } + + TARGET(LOAD_CONST) { PREDICTED(LOAD_CONST); PyObject *value = GETITEM(consts, oparg); Py_INCREF(value); @@ -1740,19 +1722,20 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_FAST): { + TARGET(STORE_FAST) { PREDICTED(STORE_FAST); PyObject *value = POP(); SETLOCAL(oparg, value); DISPATCH(); } - TARGET(LOAD_FAST__LOAD_FAST): { + TARGET(LOAD_FAST__LOAD_FAST) { PyObject *value = GETLOCAL(oparg); if (value == NULL) { goto unbound_local_error; } NEXTOPARG(); + next_instr++; Py_INCREF(value); PUSH(value); value = GETLOCAL(oparg); @@ -1764,12 +1747,13 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr NOTRACE_DISPATCH(); } - TARGET(LOAD_FAST__LOAD_CONST): { + TARGET(LOAD_FAST__LOAD_CONST) { PyObject *value = GETLOCAL(oparg); if (value == NULL) { goto unbound_local_error; } NEXTOPARG(); + next_instr++; Py_INCREF(value); PUSH(value); value = GETITEM(consts, oparg); @@ -1778,10 +1762,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr NOTRACE_DISPATCH(); } - TARGET(STORE_FAST__LOAD_FAST): { + TARGET(STORE_FAST__LOAD_FAST) { PyObject *value = POP(); SETLOCAL(oparg, value); NEXTOPARG(); + next_instr++; value = GETLOCAL(oparg); if (value == NULL) { goto unbound_local_error; @@ -1791,18 +1776,20 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr NOTRACE_DISPATCH(); } - TARGET(STORE_FAST__STORE_FAST): { + TARGET(STORE_FAST__STORE_FAST) { PyObject *value = POP(); SETLOCAL(oparg, value); NEXTOPARG(); + next_instr++; value = POP(); SETLOCAL(oparg, value); NOTRACE_DISPATCH(); } - TARGET(LOAD_CONST__LOAD_FAST): { + TARGET(LOAD_CONST__LOAD_FAST) { PyObject *value = GETITEM(consts, oparg); NEXTOPARG(); + next_instr++; Py_INCREF(value); PUSH(value); value = GETLOCAL(oparg); @@ -1814,13 +1801,13 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr NOTRACE_DISPATCH(); } - TARGET(POP_TOP): { + TARGET(POP_TOP) { PyObject *value = POP(); Py_DECREF(value); DISPATCH(); } - TARGET(ROT_TWO): { + TARGET(ROT_TWO) { PyObject *top = TOP(); PyObject *second = SECOND(); SET_TOP(second); @@ -1828,7 +1815,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(ROT_THREE): { + TARGET(ROT_THREE) { PyObject *top = TOP(); PyObject *second = SECOND(); PyObject *third = THIRD(); @@ -1838,7 +1825,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(ROT_FOUR): { + TARGET(ROT_FOUR) { PyObject *top = TOP(); PyObject *second = SECOND(); PyObject *third = THIRD(); @@ -1850,14 +1837,14 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DUP_TOP): { + TARGET(DUP_TOP) { PyObject *top = TOP(); Py_INCREF(top); PUSH(top); DISPATCH(); } - TARGET(DUP_TOP_TWO): { + TARGET(DUP_TOP_TWO) { PyObject *top = TOP(); PyObject *second = SECOND(); Py_INCREF(top); @@ -1868,7 +1855,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(UNARY_POSITIVE): { + TARGET(UNARY_POSITIVE) { PyObject *value = TOP(); PyObject *res = PyNumber_Positive(value); Py_DECREF(value); @@ -1878,7 +1865,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(UNARY_NEGATIVE): { + TARGET(UNARY_NEGATIVE) { PyObject *value = TOP(); PyObject *res = PyNumber_Negative(value); Py_DECREF(value); @@ -1888,7 +1875,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(UNARY_NOT): { + TARGET(UNARY_NOT) { PyObject *value = TOP(); int err = PyObject_IsTrue(value); Py_DECREF(value); @@ -1906,7 +1893,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } - TARGET(UNARY_INVERT): { + TARGET(UNARY_INVERT) { PyObject *value = TOP(); PyObject *res = PyNumber_Invert(value); Py_DECREF(value); @@ -1916,7 +1903,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_POWER): { + TARGET(BINARY_POWER) { PyObject *exp = POP(); PyObject *base = TOP(); PyObject *res = PyNumber_Power(base, exp, Py_None); @@ -1928,7 +1915,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_MULTIPLY): { + TARGET(BINARY_MULTIPLY) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Multiply(left, right); @@ -1940,7 +1927,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_MATRIX_MULTIPLY): { + TARGET(BINARY_MATRIX_MULTIPLY) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_MatrixMultiply(left, right); @@ -1952,7 +1939,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_TRUE_DIVIDE): { + TARGET(BINARY_TRUE_DIVIDE) { PyObject *divisor = POP(); PyObject *dividend = TOP(); PyObject *quotient = PyNumber_TrueDivide(dividend, divisor); @@ -1964,7 +1951,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_FLOOR_DIVIDE): { + TARGET(BINARY_FLOOR_DIVIDE) { PyObject *divisor = POP(); PyObject *dividend = TOP(); PyObject *quotient = PyNumber_FloorDivide(dividend, divisor); @@ -1976,7 +1963,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_MODULO): { + TARGET(BINARY_MODULO) { PyObject *divisor = POP(); PyObject *dividend = TOP(); PyObject *res; @@ -1996,7 +1983,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_ADD): { + TARGET(BINARY_ADD) { PREDICTED(BINARY_ADD); STAT_INC(BINARY_ADD, unquickened); PyObject *right = POP(); @@ -2011,7 +1998,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_ADD_ADAPTIVE): { + TARGET(BINARY_ADD_ADAPTIVE) { if (oparg == 0) { PyObject *left = SECOND(); PyObject *right = TOP(); @@ -2029,7 +2016,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(BINARY_ADD_UNICODE): { + TARGET(BINARY_ADD_UNICODE) { PyObject *left = SECOND(); PyObject *right = TOP(); DEOPT_IF(!PyUnicode_CheckExact(left), BINARY_ADD); @@ -2047,7 +2034,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_ADD_UNICODE_INPLACE_FAST): { + TARGET(BINARY_ADD_UNICODE_INPLACE_FAST) { PyObject *left = SECOND(); PyObject *right = TOP(); DEOPT_IF(!PyUnicode_CheckExact(left), BINARY_ADD); @@ -2076,7 +2063,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_ADD_FLOAT): { + TARGET(BINARY_ADD_FLOAT) { PyObject *left = SECOND(); PyObject *right = TOP(); DEOPT_IF(!PyFloat_CheckExact(left), BINARY_ADD); @@ -2096,7 +2083,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_ADD_INT): { + TARGET(BINARY_ADD_INT) { PyObject *left = SECOND(); PyObject *right = TOP(); DEOPT_IF(!PyLong_CheckExact(left), BINARY_ADD); @@ -2114,7 +2101,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_SUBTRACT): { + TARGET(BINARY_SUBTRACT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *diff = PyNumber_Subtract(left, right); @@ -2126,7 +2113,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_SUBSCR): { + TARGET(BINARY_SUBSCR) { PREDICTED(BINARY_SUBSCR); STAT_INC(BINARY_SUBSCR, unquickened); PyObject *sub = POP(); @@ -2140,7 +2127,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_SUBSCR_ADAPTIVE): { + TARGET(BINARY_SUBSCR_ADAPTIVE) { if (oparg == 0) { PyObject *sub = TOP(); PyObject *container = SECOND(); @@ -2161,7 +2148,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(BINARY_SUBSCR_LIST_INT): { + TARGET(BINARY_SUBSCR_LIST_INT) { PyObject *sub = TOP(); PyObject *list = SECOND(); DEOPT_IF(!PyLong_CheckExact(sub), BINARY_SUBSCR); @@ -2185,7 +2172,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_SUBSCR_TUPLE_INT): { + TARGET(BINARY_SUBSCR_TUPLE_INT) { PyObject *sub = TOP(); PyObject *tuple = SECOND(); DEOPT_IF(!PyLong_CheckExact(sub), BINARY_SUBSCR); @@ -2209,7 +2196,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_SUBSCR_DICT): { + TARGET(BINARY_SUBSCR_DICT) { PyObject *dict = SECOND(); DEOPT_IF(!PyDict_CheckExact(SECOND()), BINARY_SUBSCR); STAT_INC(BINARY_SUBSCR, hit); @@ -2226,7 +2213,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_LSHIFT): { + TARGET(BINARY_LSHIFT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Lshift(left, right); @@ -2238,7 +2225,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_RSHIFT): { + TARGET(BINARY_RSHIFT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Rshift(left, right); @@ -2250,7 +2237,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_AND): { + TARGET(BINARY_AND) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_And(left, right); @@ -2262,7 +2249,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_XOR): { + TARGET(BINARY_XOR) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Xor(left, right); @@ -2274,7 +2261,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BINARY_OR): { + TARGET(BINARY_OR) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Or(left, right); @@ -2286,7 +2273,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LIST_APPEND): { + TARGET(LIST_APPEND) { PyObject *v = POP(); PyObject *list = PEEK(oparg); int err; @@ -2298,7 +2285,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(SET_ADD): { + TARGET(SET_ADD) { PyObject *v = POP(); PyObject *set = PEEK(oparg); int err; @@ -2310,7 +2297,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_POWER): { + TARGET(INPLACE_POWER) { PyObject *exp = POP(); PyObject *base = TOP(); PyObject *res = PyNumber_InPlacePower(base, exp, Py_None); @@ -2322,7 +2309,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_MULTIPLY): { + TARGET(INPLACE_MULTIPLY) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceMultiply(left, right); @@ -2334,7 +2321,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_MATRIX_MULTIPLY): { + TARGET(INPLACE_MATRIX_MULTIPLY) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceMatrixMultiply(left, right); @@ -2346,7 +2333,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_TRUE_DIVIDE): { + TARGET(INPLACE_TRUE_DIVIDE) { PyObject *divisor = POP(); PyObject *dividend = TOP(); PyObject *quotient = PyNumber_InPlaceTrueDivide(dividend, divisor); @@ -2358,7 +2345,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_FLOOR_DIVIDE): { + TARGET(INPLACE_FLOOR_DIVIDE) { PyObject *divisor = POP(); PyObject *dividend = TOP(); PyObject *quotient = PyNumber_InPlaceFloorDivide(dividend, divisor); @@ -2370,7 +2357,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_MODULO): { + TARGET(INPLACE_MODULO) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *mod = PyNumber_InPlaceRemainder(left, right); @@ -2382,7 +2369,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_ADD): { + TARGET(INPLACE_ADD) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *sum; @@ -2401,7 +2388,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_SUBTRACT): { + TARGET(INPLACE_SUBTRACT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *diff = PyNumber_InPlaceSubtract(left, right); @@ -2413,7 +2400,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_LSHIFT): { + TARGET(INPLACE_LSHIFT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceLshift(left, right); @@ -2425,7 +2412,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_RSHIFT): { + TARGET(INPLACE_RSHIFT) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceRshift(left, right); @@ -2437,7 +2424,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_AND): { + TARGET(INPLACE_AND) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceAnd(left, right); @@ -2449,7 +2436,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_XOR): { + TARGET(INPLACE_XOR) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceXor(left, right); @@ -2461,7 +2448,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(INPLACE_OR): { + TARGET(INPLACE_OR) { PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_InPlaceOr(left, right); @@ -2473,7 +2460,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_SUBSCR): { + TARGET(STORE_SUBSCR) { PyObject *sub = TOP(); PyObject *container = SECOND(); PyObject *v = THIRD(); @@ -2489,7 +2476,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_SUBSCR): { + TARGET(DELETE_SUBSCR) { PyObject *sub = TOP(); PyObject *container = SECOND(); int err; @@ -2503,7 +2490,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(PRINT_EXPR): { + TARGET(PRINT_EXPR) { _Py_IDENTIFIER(displayhook); PyObject *value = POP(); PyObject *hook = _PySys_GetObjectId(&PyId_displayhook); @@ -2522,7 +2509,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(RAISE_VARARGS): { + TARGET(RAISE_VARARGS) { PyObject *cause = NULL, *exc = NULL; switch (oparg) { case 2: @@ -2544,7 +2531,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } - TARGET(RETURN_VALUE): { + TARGET(RETURN_VALUE) { retval = POP(); assert(EMPTY()); frame->f_state = FRAME_RETURNED; @@ -2552,7 +2539,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exiting; } - TARGET(GET_AITER): { + TARGET(GET_AITER) { unaryfunc getter = NULL; PyObject *iter = NULL; PyObject *obj = TOP(); @@ -2596,7 +2583,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(GET_ANEXT): { + TARGET(GET_ANEXT) { unaryfunc getter = NULL; PyObject *next_iter = NULL; PyObject *awaitable = NULL; @@ -2647,7 +2634,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(GET_AWAITABLE): { + TARGET(GET_AWAITABLE) { PREDICTED(GET_AWAITABLE); PyObject *iterable = TOP(); PyObject *iter = _PyCoro_GetAwaitableIter(iterable); @@ -2688,7 +2675,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(YIELD_FROM): { + TARGET(YIELD_FROM) { PyObject *v = POP(); PyObject *receiver = TOP(); PySendResult gen_status; @@ -2740,7 +2727,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exiting; } - TARGET(YIELD_VALUE): { + TARGET(YIELD_VALUE) { retval = POP(); if (co->co_flags & CO_ASYNC_GENERATOR) { @@ -2757,7 +2744,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exiting; } - TARGET(GEN_START): { + TARGET(GEN_START) { PyObject *none = POP(); Py_DECREF(none); if (!Py_IsNone(none)) { @@ -2781,7 +2768,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(POP_EXCEPT): { + TARGET(POP_EXCEPT) { PyObject *type, *value, *traceback; _PyErr_StackItem *exc_info; exc_info = tstate->exc_info; @@ -2797,7 +2784,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(POP_EXCEPT_AND_RERAISE): { + TARGET(POP_EXCEPT_AND_RERAISE) { PyObject *lasti = PEEK(4); if (PyLong_Check(lasti)) { frame->f_lasti = PyLong_AsLong(lasti); @@ -2827,7 +2814,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exception_unwind; } - TARGET(RERAISE): { + TARGET(RERAISE) { if (oparg) { PyObject *lasti = PEEK(oparg+3); if (PyLong_Check(lasti)) { @@ -2848,7 +2835,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exception_unwind; } - TARGET(END_ASYNC_FOR): { + TARGET(END_ASYNC_FOR) { PyObject *exc = POP(); PyObject *val = POP(); PyObject *tb = POP(); @@ -2866,14 +2853,14 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(LOAD_ASSERTION_ERROR): { + TARGET(LOAD_ASSERTION_ERROR) { PyObject *value = PyExc_AssertionError; Py_INCREF(value); PUSH(value); DISPATCH(); } - TARGET(LOAD_BUILD_CLASS): { + TARGET(LOAD_BUILD_CLASS) { _Py_IDENTIFIER(__build_class__); PyObject *bc; @@ -2904,7 +2891,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_NAME): { + TARGET(STORE_NAME) { PyObject *name = GETITEM(names, oparg); PyObject *v = POP(); PyObject *ns = LOCALS(); @@ -2925,7 +2912,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_NAME): { + TARGET(DELETE_NAME) { PyObject *name = GETITEM(names, oparg); PyObject *ns = LOCALS(); int err; @@ -2944,7 +2931,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(UNPACK_SEQUENCE): { + TARGET(UNPACK_SEQUENCE) { PREDICTED(UNPACK_SEQUENCE); PyObject *seq = POP(), *item, **items; if (PyTuple_CheckExact(seq) && @@ -2975,7 +2962,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(UNPACK_EX): { + TARGET(UNPACK_EX) { int totalargs = 1 + (oparg & 0xFF) + (oparg >> 8); PyObject *seq = POP(); @@ -2990,7 +2977,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_ATTR): { + TARGET(STORE_ATTR) { PREDICTED(STORE_ATTR); STAT_INC(STORE_ATTR, unquickened); PyObject *name = GETITEM(names, oparg); @@ -3006,7 +2993,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_ATTR): { + TARGET(DELETE_ATTR) { PyObject *name = GETITEM(names, oparg); PyObject *owner = POP(); int err; @@ -3017,7 +3004,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_GLOBAL): { + TARGET(STORE_GLOBAL) { PyObject *name = GETITEM(names, oparg); PyObject *v = POP(); int err; @@ -3028,7 +3015,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_GLOBAL): { + TARGET(DELETE_GLOBAL) { PyObject *name = GETITEM(names, oparg); int err; err = PyDict_DelItem(GLOBALS(), name); @@ -3042,7 +3029,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_NAME): { + TARGET(LOAD_NAME) { PyObject *name = GETITEM(names, oparg); PyObject *locals = LOCALS(); PyObject *v; @@ -3106,7 +3093,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_GLOBAL): { + TARGET(LOAD_GLOBAL) { PREDICTED(LOAD_GLOBAL); STAT_INC(LOAD_GLOBAL, unquickened); PyObject *name = GETITEM(names, oparg); @@ -3156,7 +3143,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_GLOBAL_ADAPTIVE): { + TARGET(LOAD_GLOBAL_ADAPTIVE) { assert(cframe.use_tracing == 0); SpecializedCacheEntry *cache = GET_CACHE(); if (cache->adaptive.counter == 0) { @@ -3176,7 +3163,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(LOAD_GLOBAL_MODULE): { + TARGET(LOAD_GLOBAL_MODULE) { assert(cframe.use_tracing == 0); DEOPT_IF(!PyDict_CheckExact(GLOBALS()), LOAD_GLOBAL); PyDictObject *dict = (PyDictObject *)GLOBALS(); @@ -3194,7 +3181,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_GLOBAL_BUILTIN): { + TARGET(LOAD_GLOBAL_BUILTIN) { assert(cframe.use_tracing == 0); DEOPT_IF(!PyDict_CheckExact(GLOBALS()), LOAD_GLOBAL); DEOPT_IF(!PyDict_CheckExact(BUILTINS()), LOAD_GLOBAL); @@ -3215,7 +3202,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_FAST): { + TARGET(DELETE_FAST) { PyObject *v = GETLOCAL(oparg); if (v != NULL) { SETLOCAL(oparg, NULL); @@ -3229,7 +3216,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } - TARGET(MAKE_CELL): { + TARGET(MAKE_CELL) { // "initial" is probably NULL but not if it's an arg (or set // via PyFrame_LocalsToFast() before MAKE_CELL has run). PyObject *initial = GETLOCAL(oparg); @@ -3241,7 +3228,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DELETE_DEREF): { + TARGET(DELETE_DEREF) { PyObject *cell = GETLOCAL(oparg); PyObject *oldobj = PyCell_GET(cell); if (oldobj != NULL) { @@ -3253,7 +3240,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } - TARGET(LOAD_CLASSDEREF): { + TARGET(LOAD_CLASSDEREF) { PyObject *name, *value, *locals = LOCALS(); assert(locals); assert(oparg >= 0 && oparg < co->co_nlocalsplus); @@ -3289,7 +3276,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_DEREF): { + TARGET(LOAD_DEREF) { PyObject *cell = GETLOCAL(oparg); PyObject *value = PyCell_GET(cell); if (value == NULL) { @@ -3301,7 +3288,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_DEREF): { + TARGET(STORE_DEREF) { PyObject *v = POP(); PyObject *cell = GETLOCAL(oparg); PyObject *oldobj = PyCell_GET(cell); @@ -3310,7 +3297,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_STRING): { + TARGET(BUILD_STRING) { PyObject *str; PyObject *empty = PyUnicode_New(0, 0); if (empty == NULL) { @@ -3328,7 +3315,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_TUPLE): { + TARGET(BUILD_TUPLE) { PyObject *tup = PyTuple_New(oparg); if (tup == NULL) goto error; @@ -3340,7 +3327,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_LIST): { + TARGET(BUILD_LIST) { PyObject *list = PyList_New(oparg); if (list == NULL) goto error; @@ -3352,7 +3339,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LIST_TO_TUPLE): { + TARGET(LIST_TO_TUPLE) { PyObject *list = POP(); PyObject *tuple = PyList_AsTuple(list); Py_DECREF(list); @@ -3363,7 +3350,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LIST_EXTEND): { + TARGET(LIST_EXTEND) { PyObject *iterable = POP(); PyObject *list = PEEK(oparg); PyObject *none_val = _PyList_Extend((PyListObject *)list, iterable); @@ -3384,7 +3371,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(SET_UPDATE): { + TARGET(SET_UPDATE) { PyObject *iterable = POP(); PyObject *set = PEEK(oparg); int err = _PySet_Update(set, iterable); @@ -3395,7 +3382,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_SET): { + TARGET(BUILD_SET) { PyObject *set = PySet_New(NULL); int err = 0; int i; @@ -3416,7 +3403,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_MAP): { + TARGET(BUILD_MAP) { Py_ssize_t i; PyObject *map = _PyDict_NewPresized((Py_ssize_t)oparg); if (map == NULL) @@ -3440,7 +3427,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(SETUP_ANNOTATIONS): { + TARGET(SETUP_ANNOTATIONS) { _Py_IDENTIFIER(__annotations__); int err; PyObject *ann_dict; @@ -3499,7 +3486,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_CONST_KEY_MAP): { + TARGET(BUILD_CONST_KEY_MAP) { Py_ssize_t i; PyObject *map; PyObject *keys = TOP(); @@ -3532,7 +3519,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DICT_UPDATE): { + TARGET(DICT_UPDATE) { PyObject *update = POP(); PyObject *dict = PEEK(oparg); if (PyDict_Update(dict, update) < 0) { @@ -3548,7 +3535,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(DICT_MERGE): { + TARGET(DICT_MERGE) { PyObject *update = POP(); PyObject *dict = PEEK(oparg); @@ -3562,7 +3549,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MAP_ADD): { + TARGET(MAP_ADD) { PyObject *value = TOP(); PyObject *key = SECOND(); PyObject *map; @@ -3579,7 +3566,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_ATTR): { + TARGET(LOAD_ATTR) { PREDICTED(LOAD_ATTR); STAT_INC(LOAD_ATTR, unquickened); PyObject *name = GETITEM(names, oparg); @@ -3593,7 +3580,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_ATTR_ADAPTIVE): { + TARGET(LOAD_ATTR_ADAPTIVE) { assert(cframe.use_tracing == 0); SpecializedCacheEntry *cache = GET_CACHE(); if (cache->adaptive.counter == 0) { @@ -3614,7 +3601,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(LOAD_ATTR_SPLIT_KEYS): { + TARGET(LOAD_ATTR_SPLIT_KEYS) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyObject *res; @@ -3639,7 +3626,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_ATTR_MODULE): { + TARGET(LOAD_ATTR_MODULE) { assert(cframe.use_tracing == 0); // shared with LOAD_METHOD_MODULE PyObject *owner = TOP(); @@ -3650,7 +3637,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_ATTR_WITH_HINT): { + TARGET(LOAD_ATTR_WITH_HINT) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyObject *res; @@ -3679,7 +3666,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_ATTR_SLOT): { + TARGET(LOAD_ATTR_SLOT) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyObject *res; @@ -3700,7 +3687,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_ATTR_ADAPTIVE): { + TARGET(STORE_ATTR_ADAPTIVE) { assert(cframe.use_tracing == 0); SpecializedCacheEntry *cache = GET_CACHE(); if (cache->adaptive.counter == 0) { @@ -3721,7 +3708,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(STORE_ATTR_SPLIT_KEYS): { + TARGET(STORE_ATTR_SPLIT_KEYS) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyTypeObject *tp = Py_TYPE(owner); @@ -3759,7 +3746,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_ATTR_WITH_HINT): { + TARGET(STORE_ATTR_WITH_HINT) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyTypeObject *tp = Py_TYPE(owner); @@ -3795,7 +3782,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(STORE_ATTR_SLOT): { + TARGET(STORE_ATTR_SLOT) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyTypeObject *tp = Py_TYPE(owner); @@ -3816,7 +3803,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(COMPARE_OP): { + TARGET(COMPARE_OP) { assert(oparg <= Py_GE); PyObject *right = POP(); PyObject *left = TOP(); @@ -3831,7 +3818,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(IS_OP): { + TARGET(IS_OP) { PyObject *right = POP(); PyObject *left = TOP(); int res = Py_Is(left, right) ^ oparg; @@ -3845,7 +3832,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CONTAINS_OP): { + TARGET(CONTAINS_OP) { PyObject *right = POP(); PyObject *left = POP(); int res = PySequence_Contains(right, left); @@ -3865,7 +3852,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr #define CANNOT_CATCH_MSG "catching classes that do not inherit from "\ "BaseException is not allowed" - TARGET(JUMP_IF_NOT_EXC_MATCH): { + TARGET(JUMP_IF_NOT_EXC_MATCH) { PyObject *right = POP(); PyObject *left = POP(); if (PyTuple_Check(right)) { @@ -3906,7 +3893,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(IMPORT_NAME): { + TARGET(IMPORT_NAME) { PyObject *name = GETITEM(names, oparg); PyObject *fromlist = POP(); PyObject *level = TOP(); @@ -3920,7 +3907,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(IMPORT_STAR): { + TARGET(IMPORT_STAR) { PyObject *from = POP(), *locals; int err; if (_PyFrame_FastToLocalsWithError(frame) < 0) { @@ -3943,7 +3930,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(IMPORT_FROM): { + TARGET(IMPORT_FROM) { PyObject *name = GETITEM(names, oparg); PyObject *from = TOP(); PyObject *res; @@ -3954,12 +3941,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(JUMP_FORWARD): { + TARGET(JUMP_FORWARD) { JUMPBY(oparg); DISPATCH(); } - TARGET(POP_JUMP_IF_FALSE): { + TARGET(POP_JUMP_IF_FALSE) { PREDICTED(POP_JUMP_IF_FALSE); PyObject *cond = POP(); int err; @@ -3986,7 +3973,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(POP_JUMP_IF_TRUE): { + TARGET(POP_JUMP_IF_TRUE) { PREDICTED(POP_JUMP_IF_TRUE); PyObject *cond = POP(); int err; @@ -4013,7 +4000,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(JUMP_IF_FALSE_OR_POP): { + TARGET(JUMP_IF_FALSE_OR_POP) { PyObject *cond = TOP(); int err; if (Py_IsTrue(cond)) { @@ -4037,7 +4024,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(JUMP_IF_TRUE_OR_POP): { + TARGET(JUMP_IF_TRUE_OR_POP) { PyObject *cond = TOP(); int err; if (Py_IsFalse(cond)) { @@ -4062,7 +4049,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(JUMP_ABSOLUTE): { + TARGET(JUMP_ABSOLUTE) { PREDICTED(JUMP_ABSOLUTE); if (oparg < INSTR_OFFSET()) { /* Increment the warmup counter and quicken if warm enough @@ -4084,13 +4071,13 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(JUMP_ABSOLUTE_QUICK): { + TARGET(JUMP_ABSOLUTE_QUICK) { JUMPTO(oparg); CHECK_EVAL_BREAKER(); DISPATCH(); } - TARGET(GET_LEN): { + TARGET(GET_LEN) { // PUSH(len(TOS)) Py_ssize_t len_i = PyObject_Length(TOP()); if (len_i < 0) { @@ -4104,7 +4091,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MATCH_CLASS): { + TARGET(MATCH_CLASS) { // Pop TOS. On success, set TOS to True and TOS1 to a tuple of // attributes. On failure, set TOS to False. PyObject *names = POP(); @@ -4127,7 +4114,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MATCH_MAPPING): { + TARGET(MATCH_MAPPING) { PyObject *subject = TOP(); int match = Py_TYPE(subject)->tp_flags & Py_TPFLAGS_MAPPING; PyObject *res = match ? Py_True : Py_False; @@ -4136,7 +4123,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MATCH_SEQUENCE): { + TARGET(MATCH_SEQUENCE) { PyObject *subject = TOP(); int match = Py_TYPE(subject)->tp_flags & Py_TPFLAGS_SEQUENCE; PyObject *res = match ? Py_True : Py_False; @@ -4145,7 +4132,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MATCH_KEYS): { + TARGET(MATCH_KEYS) { // On successful match for all keys, PUSH(values) and PUSH(True). // Otherwise, PUSH(None) and PUSH(False). PyObject *keys = TOP(); @@ -4166,7 +4153,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(COPY_DICT_WITHOUT_KEYS): { + TARGET(COPY_DICT_WITHOUT_KEYS) { // rest = dict(TOS1) // for key in TOS: // del rest[key] @@ -4192,7 +4179,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(GET_ITER): { + TARGET(GET_ITER) { /* before: [obj]; after [getiter(obj)] */ PyObject *iterable = TOP(); PyObject *iter = PyObject_GetIter(iterable); @@ -4205,7 +4192,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(GET_YIELD_FROM_ITER): { + TARGET(GET_YIELD_FROM_ITER) { /* before: [obj]; after [getiter(obj)] */ PyObject *iterable = TOP(); PyObject *iter; @@ -4234,7 +4221,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(FOR_ITER): { + TARGET(FOR_ITER) { PREDICTED(FOR_ITER); /* before: [iter]; after: [iter, iter()] *or* [] */ PyObject *iter = TOP(); @@ -4261,7 +4248,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BEFORE_ASYNC_WITH): { + TARGET(BEFORE_ASYNC_WITH) { _Py_IDENTIFIER(__aenter__); _Py_IDENTIFIER(__aexit__); PyObject *mgr = TOP(); @@ -4299,7 +4286,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BEFORE_WITH): { + TARGET(BEFORE_WITH) { _Py_IDENTIFIER(__enter__); _Py_IDENTIFIER(__exit__); PyObject *mgr = TOP(); @@ -4337,7 +4324,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(WITH_EXCEPT_START): { + TARGET(WITH_EXCEPT_START) { /* At the top of the stack are 8 values: - (TOP, SECOND, THIRD) = exc_info() - (FOURTH, FIFTH, SIXTH) = previous exception @@ -4367,7 +4354,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(PUSH_EXC_INFO): { + TARGET(PUSH_EXC_INFO) { PyObject *type = TOP(); PyObject *value = SECOND(); PyObject *tb = THIRD(); @@ -4398,7 +4385,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_METHOD): { + TARGET(LOAD_METHOD) { PREDICTED(LOAD_METHOD); STAT_INC(LOAD_METHOD, unquickened); /* Designed to work in tandem with CALL_METHOD. */ @@ -4437,7 +4424,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_METHOD_ADAPTIVE): { + TARGET(LOAD_METHOD_ADAPTIVE) { assert(cframe.use_tracing == 0); SpecializedCacheEntry *cache = GET_CACHE(); if (cache->adaptive.counter == 0) { @@ -4458,7 +4445,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(LOAD_METHOD_CACHED): { + TARGET(LOAD_METHOD_CACHED) { /* LOAD_METHOD, with cached method object */ assert(cframe.use_tracing == 0); PyObject *self = TOP(); @@ -4495,7 +4482,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_METHOD_MODULE): { + TARGET(LOAD_METHOD_MODULE) { /* LOAD_METHOD, for module methods */ assert(cframe.use_tracing == 0); PyObject *owner = TOP(); @@ -4507,7 +4494,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(LOAD_METHOD_CLASS): { + TARGET(LOAD_METHOD_CLASS) { /* LOAD_METHOD, for class methods */ assert(cframe.use_tracing == 0); SpecializedCacheEntry *caches = GET_CACHE(); @@ -4533,7 +4520,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CALL_METHOD): { + TARGET(CALL_METHOD) { /* Designed to work in tamdem with LOAD_METHOD. */ PyObject **sp, *res; int meth_found; @@ -4577,7 +4564,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CALL_METHOD_KW): { + TARGET(CALL_METHOD_KW) { /* Designed to work in tandem with LOAD_METHOD. Same as CALL_METHOD but pops TOS to get a tuple of keyword names. */ PyObject **sp, *res; @@ -4601,7 +4588,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CALL_FUNCTION): { + TARGET(CALL_FUNCTION) { PREDICTED(CALL_FUNCTION); PyObject **sp, *res; sp = stack_pointer; @@ -4615,7 +4602,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CALL_FUNCTION_KW): { + TARGET(CALL_FUNCTION_KW) { PyObject **sp, *res, *names; names = POP(); @@ -4635,7 +4622,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(CALL_FUNCTION_EX): { + TARGET(CALL_FUNCTION_EX) { PREDICTED(CALL_FUNCTION_EX); PyObject *func, *callargs, *kwargs = NULL, *result; if (oparg & 0x01) { @@ -4682,7 +4669,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(MAKE_FUNCTION): { + TARGET(MAKE_FUNCTION) { PyObject *codeobj = POP(); PyFunctionObject *func = (PyFunctionObject *) PyFunction_New(codeobj, GLOBALS()); @@ -4713,7 +4700,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(BUILD_SLICE): { + TARGET(BUILD_SLICE) { PyObject *start, *stop, *step, *slice; if (oparg == 3) step = POP(); @@ -4731,7 +4718,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(FORMAT_VALUE): { + TARGET(FORMAT_VALUE) { /* Handles f-string value formatting. */ PyObject *result; PyObject *fmt_spec; @@ -4791,7 +4778,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(ROT_N): { + TARGET(ROT_N) { PyObject *top = TOP(); memmove(&PEEK(oparg - 1), &PEEK(oparg), sizeof(PyObject*) * (oparg - 1)); @@ -4799,7 +4786,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - TARGET(EXTENDED_ARG): { + TARGET(EXTENDED_ARG) { int oldoparg = oparg; NEXTOPARG(); oparg |= oldoparg << 8; @@ -4807,6 +4794,45 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH_GOTO(); } +#if USE_COMPUTED_GOTOS + TARGET_DO_TRACING: { +#else + case DO_TRACING: { +#endif + int instr_prev = skip_backwards_over_extended_args(co, frame->f_lasti); + frame->f_lasti = INSTR_OFFSET(); + TRACING_NEXTOPARG(); + if (PyDTrace_LINE_ENABLED()) { + maybe_dtrace_line(frame, &tstate->trace_info, instr_prev); + } + /* line-by-line tracing support */ + + if (cframe.use_tracing && + tstate->c_tracefunc != NULL && !tstate->tracing) { + int err; + /* see maybe_call_line_trace() + for expository comments */ + _PyFrame_SetStackPointer(frame, stack_pointer); + + err = maybe_call_line_trace(tstate->c_tracefunc, + tstate->c_traceobj, + tstate, frame, instr_prev); + if (err) { + /* trace function raised an exception */ + next_instr++; + goto error; + } + /* Reload possibly changed frame fields */ + JUMPTO(frame->f_lasti); + + stack_pointer = _PyFrame_GetStackPointer(frame); + frame->stacktop = -1; + TRACING_NEXTOPARG(); + } + PRE_DISPATCH_GOTO(); + DISPATCH_GOTO(); + } + #if USE_COMPUTED_GOTOS _unknown_opcode: @@ -6001,7 +6027,7 @@ call_trace(Py_tracefunc func, PyObject *obj, result = func(obj, f, what, arg); f->f_lineno = 0; tstate->cframe->use_tracing = ((tstate->c_tracefunc != NULL) - || (tstate->c_profilefunc != NULL)); + || (tstate->c_profilefunc != NULL)) ? 255 : 0; tstate->tracing--; return result; } @@ -6016,7 +6042,7 @@ _PyEval_CallTracing(PyObject *func, PyObject *args) tstate->tracing = 0; tstate->cframe->use_tracing = ((tstate->c_tracefunc != NULL) - || (tstate->c_profilefunc != NULL)); + || (tstate->c_profilefunc != NULL)) ? 255 : 0; result = PyObject_Call(func, args, NULL); tstate->tracing = save_tracing; tstate->cframe->use_tracing = save_use_tracing; @@ -6081,7 +6107,7 @@ _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_profilefunc = func; /* Flag that tracing or profiling is turned on */ - tstate->cframe->use_tracing = (func != NULL) || (tstate->c_tracefunc != NULL); + tstate->cframe->use_tracing = (func != NULL) || (tstate->c_tracefunc != NULL) ? 255 : 0; return 0; } @@ -6114,7 +6140,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_tracefunc = NULL; tstate->c_traceobj = NULL; /* Must make sure that profiling is not ignored if 'traceobj' is freed */ - tstate->cframe->use_tracing = (tstate->c_profilefunc != NULL); + tstate->cframe->use_tracing = (tstate->c_profilefunc != NULL) ? 255 : 0; Py_XDECREF(traceobj); Py_XINCREF(arg); @@ -6123,7 +6149,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) /* Flag that tracing or profiling is turned on */ tstate->cframe->use_tracing = ((func != NULL) - || (tstate->c_profilefunc != NULL)); + || (tstate->c_profilefunc != NULL)) ? 255 : 0; return 0; } @@ -6871,6 +6897,7 @@ unicode_concatenate(PyThreadState *tstate, PyObject *v, PyObject *w, */ int opcode, oparg; NEXTOPARG(); + next_instr++; switch (opcode) { case STORE_FAST: { diff --git a/Python/makeopcodetargets.py b/Python/makeopcodetargets.py index 189d72a8c84af..3bf2e35ccb6da 100755 --- a/Python/makeopcodetargets.py +++ b/Python/makeopcodetargets.py @@ -32,6 +32,7 @@ def write_contents(f): """ opcode = find_module('opcode') targets = ['_unknown_opcode'] * 256 + targets[255] = "TARGET_DO_TRACING" for opname, op in opcode.opmap.items(): targets[op] = "TARGET_%s" % opname next_op = 1 diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index f3bfae545bcd4..30df68382536a 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -254,5 +254,5 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode + &&TARGET_DO_TRACING }; diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 6e7e45bf3fde2..f2cd3a9d5a010 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -268,7 +268,7 @@ sys_audit_tstate(PyThreadState *ts, const char *event, break; } if (canTrace) { - ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc); + ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc) ? 255 : 0; ts->tracing--; } PyObject* args[2] = {eventName, eventArgs}; @@ -283,7 +283,7 @@ sys_audit_tstate(PyThreadState *ts, const char *event, Py_DECREF(o); Py_CLEAR(hook); } - ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc); + ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc) ? 255 : 0; ts->tracing--; if (_PyErr_Occurred(ts)) { goto exit; diff --git a/Tools/scripts/generate_opcode_h.py b/Tools/scripts/generate_opcode_h.py index 48875d2a9dd39..dca9bbcfcbc83 100644 --- a/Tools/scripts/generate_opcode_h.py +++ b/Tools/scripts/generate_opcode_h.py @@ -72,7 +72,7 @@ def main(opcode_py, outfile='Include/opcode.h'): next_op += 1 fobj.write("#define %-23s %3s\n" % (name, next_op)) used[next_op] = True - + fobj.write("#define DO_TRACING 255\n") fobj.write("#ifdef NEED_OPCODE_JUMP_TABLES\n") write_int_array_from_ops("_PyOpcode_RelativeJump", opcode['hasjrel'], fobj) write_int_array_from_ops("_PyOpcode_Jump", opcode['hasjrel'] + opcode['hasjabs'], fobj) From webhook-mailer at python.org Tue Oct 5 06:03:02 2021 From: webhook-mailer at python.org (pablogsal) Date: Tue, 05 Oct 2021 10:03:02 -0000 Subject: [Python-checkins] bpo-45163: Restrict added libnetwork check to builds on Haiku. (GH-28729) Message-ID: https://github.com/python/cpython/commit/0af08f343a7b792f527b78e2a35d9453039940c2 commit: 0af08f343a7b792f527b78e2a35d9453039940c2 branch: main author: Ned Deily committer: pablogsal date: 2021-10-05T11:02:57+01:00 summary: bpo-45163: Restrict added libnetwork check to builds on Haiku. (GH-28729) For example, without the guard the check could cause macOS installer builds to fail to install on older supported macOS releases where libnetwork is not available and is not needed on any release. files: M configure M configure.ac diff --git a/configure b/configure index 3b39641bab37c..3a6cf305171bc 100755 --- a/configure +++ b/configure @@ -10573,8 +10573,9 @@ if test "x$ac_cv_lib_socket_socket" = xyes; then : fi # SVR4 sockets -# Haiku system library -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lnetwork" >&5 +case $ac_sys_system/$ac_sys_release in + Haiku*) + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for socket in -lnetwork" >&5 $as_echo_n "checking for socket in -lnetwork... " >&6; } if ${ac_cv_lib_network_socket+:} false; then : $as_echo_n "(cached) " >&6 @@ -10614,6 +10615,8 @@ if test "x$ac_cv_lib_network_socket" = xyes; then : LIBS="-lnetwork $LIBS" fi + ;; +esac { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-libs" >&5 $as_echo_n "checking for --with-libs... " >&6; } diff --git a/configure.ac b/configure.ac index 908dd28e7aaca..c7cb797e8f3a0 100644 --- a/configure.ac +++ b/configure.ac @@ -3099,8 +3099,11 @@ AC_SUBST(TZPATH) AC_CHECK_LIB(nsl, t_open, [LIBS="-lnsl $LIBS"]) # SVR4 AC_CHECK_LIB(socket, socket, [LIBS="-lsocket $LIBS"], [], $LIBS) # SVR4 sockets -# Haiku system library -AC_CHECK_LIB(network, socket, [LIBS="-lnetwork $LIBS"], [], $LIBS) +case $ac_sys_system/$ac_sys_release in + Haiku*) + AC_CHECK_LIB(network, socket, [LIBS="-lnetwork $LIBS"], [], $LIBS) + ;; +esac AC_MSG_CHECKING(for --with-libs) AC_ARG_WITH(libs, From webhook-mailer at python.org Tue Oct 5 06:04:56 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 10:04:56 -0000 Subject: [Python-checkins] [3.10] bpo-45371: Fix distutils' rpath support for clang (GH-28732) (GH-28733) Message-ID: https://github.com/python/cpython/commit/3733dddecaa332966e200718d4993545f8e52247 commit: 3733dddecaa332966e200718d4993545f8e52247 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-05T03:04:52-07:00 summary: [3.10] bpo-45371: Fix distutils' rpath support for clang (GH-28732) (GH-28733) Signed-off-by: Christian Heimes (cherry picked from commit ef6196028f966f22d82930b66e1371e75c5df2f7) Co-authored-by: Christian Heimes Automerge-Triggered-By: GH:tiran files: A Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst M Lib/distutils/unixccompiler.py diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py index f0792de74a1a4..d00c48981eb6d 100644 --- a/Lib/distutils/unixccompiler.py +++ b/Lib/distutils/unixccompiler.py @@ -215,7 +215,8 @@ def library_dir_option(self, dir): return "-L" + dir def _is_gcc(self, compiler_name): - return "gcc" in compiler_name or "g++" in compiler_name + # clang uses same syntax for rpath as gcc + return any(name in compiler_name for name in ("gcc", "g++", "clang")) def runtime_library_dir_option(self, dir): # XXX Hackish, at the very least. See Python bug #445902: diff --git a/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst new file mode 100644 index 0000000000000..045489be81a19 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst @@ -0,0 +1,3 @@ +Fix clang rpath issue in :mod:`distutils`. The UnixCCompiler now uses +correct clang option to add a runtime library directory (rpath) to a shared +library. From webhook-mailer at python.org Tue Oct 5 06:09:25 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 10:09:25 -0000 Subject: [Python-checkins] bpo-45371: Fix distutils' rpath support for clang (GH-28732) Message-ID: https://github.com/python/cpython/commit/3ce5e07e9a4b78302b69f898527396ff7b5fd448 commit: 3ce5e07e9a4b78302b69f898527396ff7b5fd448 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-05T03:09:21-07:00 summary: bpo-45371: Fix distutils' rpath support for clang (GH-28732) Signed-off-by: Christian Heimes (cherry picked from commit ef6196028f966f22d82930b66e1371e75c5df2f7) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst M Lib/distutils/unixccompiler.py diff --git a/Lib/distutils/unixccompiler.py b/Lib/distutils/unixccompiler.py index f0792de74a1a4..d00c48981eb6d 100644 --- a/Lib/distutils/unixccompiler.py +++ b/Lib/distutils/unixccompiler.py @@ -215,7 +215,8 @@ def library_dir_option(self, dir): return "-L" + dir def _is_gcc(self, compiler_name): - return "gcc" in compiler_name or "g++" in compiler_name + # clang uses same syntax for rpath as gcc + return any(name in compiler_name for name in ("gcc", "g++", "clang")) def runtime_library_dir_option(self, dir): # XXX Hackish, at the very least. See Python bug #445902: diff --git a/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst new file mode 100644 index 0000000000000..045489be81a19 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst @@ -0,0 +1,3 @@ +Fix clang rpath issue in :mod:`distutils`. The UnixCCompiler now uses +correct clang option to add a runtime library directory (rpath) to a shared +library. From webhook-mailer at python.org Tue Oct 5 08:17:21 2021 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Oct 2021 12:17:21 -0000 Subject: [Python-checkins] bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) Message-ID: https://github.com/python/cpython/commit/de4052fe0633e3a053e66c8477f13677054d6ede commit: de4052fe0633e3a053e66c8477f13677054d6ede branch: main author: Jeremy Kloth committer: zooba date: 2021-10-05T13:17:13+01:00 summary: bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) files: M Lib/test/test_winconsoleio.py diff --git a/Lib/test/test_winconsoleio.py b/Lib/test/test_winconsoleio.py index 1807e47c66c38..70a85552cc03b 100644 --- a/Lib/test/test_winconsoleio.py +++ b/Lib/test/test_winconsoleio.py @@ -92,9 +92,11 @@ def test_open_name(self): f.close() f.close() - f = open('C:/con', 'rb', buffering=0) - self.assertIsInstance(f, ConIO) - f.close() + # bpo-45354: Windows 11 changed MS-DOS device name handling + if sys.getwindowsversion()[:3] < (10, 0, 22000): + f = open('C:/con', 'rb', buffering=0) + self.assertIsInstance(f, ConIO) + f.close() @unittest.skipIf(sys.getwindowsversion()[:2] <= (6, 1), "test does not work on Windows 7 and earlier") @@ -114,7 +116,8 @@ def test_conout_path(self): conout_path = os.path.join(temp_path, 'CONOUT$') with open(conout_path, 'wb', buffering=0) as f: - if sys.getwindowsversion()[:2] > (6, 1): + # bpo-45354: Windows 11 changed MS-DOS device name handling + if (6, 1) < sys.getwindowsversion()[:3] < (10, 0, 22000): self.assertIsInstance(f, ConIO) else: self.assertNotIsInstance(f, ConIO) From webhook-mailer at python.org Tue Oct 5 08:37:55 2021 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Oct 2021 12:37:55 -0000 Subject: [Python-checkins] bpo-45375: Fix assertion failure due to searching for stdlib in unnormalised paths (GH-28735) Message-ID: https://github.com/python/cpython/commit/5146877623ebe8a2806411703b0de9c0aba179a1 commit: 5146877623ebe8a2806411703b0de9c0aba179a1 branch: main author: Steve Dower committer: zooba date: 2021-10-05T13:37:43+01:00 summary: bpo-45375: Fix assertion failure due to searching for stdlib in unnormalised paths (GH-28735) files: A Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst M PC/getpathp.c M PCbuild/regen.targets diff --git a/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst b/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst new file mode 100644 index 0000000000000..c72164373abe6 --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst @@ -0,0 +1,2 @@ +Fixes an assertion failure due to searching for the standard library in +unnormalised paths. diff --git a/PC/getpathp.c b/PC/getpathp.c index 16bb4997f819b..98a754976c670 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -265,7 +265,21 @@ canonicalize(wchar_t *buffer, const wchar_t *path) return _PyStatus_NO_MEMORY(); } - if (FAILED(PathCchCanonicalizeEx(buffer, MAXPATHLEN + 1, path, 0))) { + if (PathIsRelativeW(path)) { + wchar_t buff[MAXPATHLEN]; + if (!GetCurrentDirectoryW(MAXPATHLEN, buff)) { + return _PyStatus_ERR("unable to find current working directory"); + } + if (FAILED(PathCchCombineEx(buff, MAXPATHLEN + 1, buff, path, PATHCCH_ALLOW_LONG_PATHS))) { + return INIT_ERR_BUFFER_OVERFLOW(); + } + if (FAILED(PathCchCanonicalizeEx(buffer, MAXPATHLEN + 1, buff, PATHCCH_ALLOW_LONG_PATHS))) { + return INIT_ERR_BUFFER_OVERFLOW(); + } + return _PyStatus_OK(); + } + + if (FAILED(PathCchCanonicalizeEx(buffer, MAXPATHLEN + 1, path, PATHCCH_ALLOW_LONG_PATHS))) { return INIT_ERR_BUFFER_OVERFLOW(); } return _PyStatus_OK(); @@ -291,6 +305,9 @@ search_for_prefix(wchar_t *prefix, const wchar_t *argv0_path) /* Search from argv0_path, until LANDMARK is found. We guarantee 'prefix' is null terminated in bounds. */ wcscpy_s(prefix, MAXPATHLEN+1, argv0_path); + if (!prefix[0]) { + return 0; + } wchar_t stdlibdir[MAXPATHLEN+1]; wcscpy_s(stdlibdir, Py_ARRAY_LENGTH(stdlibdir), prefix); /* We initialize with the longest possible path, in case it doesn't fit. @@ -925,6 +942,7 @@ calculate_module_search_path(PyCalculatePath *calculate, the parent of that. */ if (prefix[0] == L'\0') { + PyStatus status; wchar_t lookBuf[MAXPATHLEN+1]; const wchar_t *look = buf - 1; /* 'buf' is at the end of the buffer */ while (1) { @@ -939,6 +957,10 @@ calculate_module_search_path(PyCalculatePath *calculate, nchars = lookEnd-look; wcsncpy(lookBuf, look+1, nchars); lookBuf[nchars] = L'\0'; + status = canonicalize(lookBuf, lookBuf); + if (_PyStatus_EXCEPTION(status)) { + return status; + } /* Up one level to the parent */ reduce(lookBuf); if (search_for_prefix(prefix, lookBuf)) { diff --git a/PCbuild/regen.targets b/PCbuild/regen.targets index 9492cff2d8c3e..c0bde1ec6ba51 100644 --- a/PCbuild/regen.targets +++ b/PCbuild/regen.targets @@ -104,7 +104,9 @@ Condition="($(Platform) == 'Win32' or $(Platform) == 'x64') and $(Configuration) != 'PGInstrument' and $(Configuration) != 'PGUpdate'"> - From webhook-mailer at python.org Tue Oct 5 08:38:55 2021 From: webhook-mailer at python.org (zooba) Date: Tue, 05 Oct 2021 12:38:55 -0000 Subject: [Python-checkins] bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) Message-ID: https://github.com/python/cpython/commit/d0d0909a3a0b553826d1ddbb04a676fdabb61359 commit: d0d0909a3a0b553826d1ddbb04a676fdabb61359 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: zooba date: 2021-10-05T13:38:50+01:00 summary: bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) (cherry picked from commit de4052fe0633e3a053e66c8477f13677054d6ede) Co-authored-by: Jeremy Kloth files: M Lib/test/test_winconsoleio.py diff --git a/Lib/test/test_winconsoleio.py b/Lib/test/test_winconsoleio.py index 1807e47c66c38..70a85552cc03b 100644 --- a/Lib/test/test_winconsoleio.py +++ b/Lib/test/test_winconsoleio.py @@ -92,9 +92,11 @@ def test_open_name(self): f.close() f.close() - f = open('C:/con', 'rb', buffering=0) - self.assertIsInstance(f, ConIO) - f.close() + # bpo-45354: Windows 11 changed MS-DOS device name handling + if sys.getwindowsversion()[:3] < (10, 0, 22000): + f = open('C:/con', 'rb', buffering=0) + self.assertIsInstance(f, ConIO) + f.close() @unittest.skipIf(sys.getwindowsversion()[:2] <= (6, 1), "test does not work on Windows 7 and earlier") @@ -114,7 +116,8 @@ def test_conout_path(self): conout_path = os.path.join(temp_path, 'CONOUT$') with open(conout_path, 'wb', buffering=0) as f: - if sys.getwindowsversion()[:2] > (6, 1): + # bpo-45354: Windows 11 changed MS-DOS device name handling + if (6, 1) < sys.getwindowsversion()[:3] < (10, 0, 22000): self.assertIsInstance(f, ConIO) else: self.assertNotIsInstance(f, ConIO) From webhook-mailer at python.org Tue Oct 5 08:39:22 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 12:39:22 -0000 Subject: [Python-checkins] bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) Message-ID: https://github.com/python/cpython/commit/63c9a6cc8b48740c88199b5150c9948b1a61756b commit: 63c9a6cc8b48740c88199b5150c9948b1a61756b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-05T05:39:18-07:00 summary: bpo-45354: Skip obsolete device name tests on Windows 11 (GH-28712) (cherry picked from commit de4052fe0633e3a053e66c8477f13677054d6ede) Co-authored-by: Jeremy Kloth files: M Lib/test/test_winconsoleio.py diff --git a/Lib/test/test_winconsoleio.py b/Lib/test/test_winconsoleio.py index a44f7bbd27b70..7b2bfda339f95 100644 --- a/Lib/test/test_winconsoleio.py +++ b/Lib/test/test_winconsoleio.py @@ -92,9 +92,11 @@ def test_open_name(self): f.close() f.close() - f = open('C:/con', 'rb', buffering=0) - self.assertIsInstance(f, ConIO) - f.close() + # bpo-45354: Windows 11 changed MS-DOS device name handling + if sys.getwindowsversion()[:3] < (10, 0, 22000): + f = open('C:/con', 'rb', buffering=0) + self.assertIsInstance(f, ConIO) + f.close() @unittest.skipIf(sys.getwindowsversion()[:2] <= (6, 1), "test does not work on Windows 7 and earlier") @@ -114,7 +116,8 @@ def test_conout_path(self): conout_path = os.path.join(temp_path, 'CONOUT$') with open(conout_path, 'wb', buffering=0) as f: - if sys.getwindowsversion()[:2] > (6, 1): + # bpo-45354: Windows 11 changed MS-DOS device name handling + if (6, 1) < sys.getwindowsversion()[:3] < (10, 0, 22000): self.assertIsInstance(f, ConIO) else: self.assertNotIsInstance(f, ConIO) From webhook-mailer at python.org Tue Oct 5 09:19:40 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 05 Oct 2021 13:19:40 -0000 Subject: [Python-checkins] bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) Message-ID: https://github.com/python/cpython/commit/b9bb74871b27d9226df2dd3fce9d42bda8b43c2b commit: b9bb74871b27d9226df2dd3fce9d42bda8b43c2b branch: main author: Hai Shi committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-05T06:19:32-07:00 summary: bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) Automerge-Triggered-By: GH:encukou files: A Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst M Lib/test/test_capi.py M Modules/_testmultiphase.c M Python/import.c diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index db029ef731c38..bdb8f768fc313 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -766,6 +766,37 @@ def test_mutate_exception(self): self.assertFalse(hasattr(binascii.Error, "foobar")) + def test_module_state_shared_in_global(self): + """ + bpo-44050: Extension module state should be shared between interpreters + when it doesn't support sub-interpreters. + """ + r, w = os.pipe() + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + + script = textwrap.dedent(f""" + import importlib.machinery + import importlib.util + import os + + fullname = '_test_module_state_shared' + origin = importlib.util.find_spec('_testmultiphase').origin + loader = importlib.machinery.ExtensionFileLoader(fullname, origin) + spec = importlib.util.spec_from_loader(fullname, loader) + module = importlib.util.module_from_spec(spec) + attr_id = str(id(module.Error)).encode() + + os.write({w}, attr_id) + """) + exec(script) + main_attr_id = os.read(r, 100) + + ret = support.run_in_subinterp(script) + self.assertEqual(ret, 0) + subinterp_attr_id = os.read(r, 100) + self.assertEqual(main_attr_id, subinterp_attr_id) + class TestThreadState(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst new file mode 100644 index 0000000000000..d6eed9f1bcfe9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst @@ -0,0 +1,3 @@ +Extensions that indicate they use global state (by setting ``m_size`` to -1) +can again be used in multiple interpreters. This reverts to behavior of +Python 3.8. diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index ad60f32f7e7a6..e0ed77d265cdc 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -844,6 +844,28 @@ PyInit__testmultiphase_meth_state_access(PyObject *spec) return PyModuleDef_Init(&def_meth_state_access); } +static PyModuleDef def_module_state_shared = { + PyModuleDef_HEAD_INIT, + .m_name = "_test_module_state_shared", + .m_doc = PyDoc_STR("Regression Test module for single-phase init."), + .m_size = -1, +}; + +PyMODINIT_FUNC +PyInit__test_module_state_shared(PyObject *spec) +{ + PyObject *module = PyModule_Create(&def_module_state_shared); + if (module == NULL) { + return NULL; + } + + if (PyModule_AddObjectRef(module, "Error", PyExc_Exception) < 0) { + Py_DECREF(module); + return NULL; + } + return module; +} + /*** Helper for imp test ***/ diff --git a/Python/import.c b/Python/import.c index 317a836617c51..d7f126784192b 100644 --- a/Python/import.c +++ b/Python/import.c @@ -442,7 +442,9 @@ _PyImport_FixupExtensionObject(PyObject *mod, PyObject *name, return -1; } - if (_Py_IsMainInterpreter(tstate->interp)) { + // bpo-44050: Extensions and def->m_base.m_copy can be updated + // when the extension module doesn't support sub-interpreters. + if (_Py_IsMainInterpreter(tstate->interp) || def->m_size == -1) { if (def->m_size == -1) { if (def->m_base.m_copy) { /* Somebody already imported the module, From webhook-mailer at python.org Tue Oct 5 12:01:36 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Tue, 05 Oct 2021 16:01:36 -0000 Subject: [Python-checkins] bpo-45324: Capture data in FrozenImporter.find_spec() to use in exec_module(). (gh-28633) Message-ID: https://github.com/python/cpython/commit/c3d9ac8b340fcbf54cee865737e67f11fcd70ed3 commit: c3d9ac8b340fcbf54cee865737e67f11fcd70ed3 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-05T10:01:27-06:00 summary: bpo-45324: Capture data in FrozenImporter.find_spec() to use in exec_module(). (gh-28633) Before this change we end up duplicating effort and throwing away data in FrozenImporter.find_spec(). Now we do the work once in find_spec() and the only thing we do in FrozenImporter.exec_module() is turn the raw frozen data into a code object and then exec it. We've added _imp.find_frozen(), add an arg to _imp.get_frozen_object(), and updated FrozenImporter. We've also moved some code around to reduce duplication, get a little more consistency in outcomes, and be more efficient. Note that this change is mostly necessary if we want to set __file__ on frozen stdlib modules. (See https://bugs.python.org/issue21736.) https://bugs.python.org/issue45324 files: A Misc/NEWS.d/next/Core and Builtins/2021-09-29-12-02-39.bpo-45324.BTQElX.rst M Lib/importlib/_bootstrap.py M Lib/test/test_importlib/frozen/test_finder.py M Lib/test/test_importlib/frozen/test_loader.py M Python/clinic/import.c.h M Python/import.c diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index e64f7ad151755..5807577c74bce 100644 --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -826,10 +826,15 @@ def module_repr(m): @classmethod def find_spec(cls, fullname, path=None, target=None): - if _imp.is_frozen(fullname): - return spec_from_loader(fullname, cls, origin=cls._ORIGIN) - else: + info = _call_with_frames_removed(_imp.find_frozen, fullname) + if info is None: return None + data, ispkg = info + spec = spec_from_loader(fullname, cls, + origin=cls._ORIGIN, + is_package=ispkg) + spec.loader_state = data + return spec @classmethod def find_module(cls, fullname, path=None): @@ -849,11 +854,22 @@ def create_module(spec): @staticmethod def exec_module(module): - name = module.__spec__.name - if not _imp.is_frozen(name): - raise ImportError('{!r} is not a frozen module'.format(name), - name=name) - code = _call_with_frames_removed(_imp.get_frozen_object, name) + spec = module.__spec__ + name = spec.name + try: + data = spec.loader_state + except AttributeError: + if not _imp.is_frozen(name): + raise ImportError('{!r} is not a frozen module'.format(name), + name=name) + data = None + else: + # We clear the extra data we got from the finder, to save memory. + # Note that if this method is called again (e.g. by + # importlib.reload()) then _imp.get_frozen_object() will notice + # no data was provided and will look it up. + spec.loader_state = None + code = _call_with_frames_removed(_imp.get_frozen_object, name, data) exec(code, module.__dict__) @classmethod diff --git a/Lib/test/test_importlib/frozen/test_finder.py b/Lib/test/test_importlib/frozen/test_finder.py index 7d43fb0ff3e11..23d1bf7fb7773 100644 --- a/Lib/test/test_importlib/frozen/test_finder.py +++ b/Lib/test/test_importlib/frozen/test_finder.py @@ -1,13 +1,15 @@ from .. import abc -import os.path from .. import util machinery = util.import_importlib('importlib.machinery') +import _imp +import marshal +import os.path import unittest import warnings -from test.support import import_helper +from test.support import import_helper, REPO_ROOT class FindSpecTests(abc.FinderTests): @@ -19,39 +21,67 @@ def find(self, name, **kwargs): with import_helper.frozen_modules(): return finder.find_spec(name, **kwargs) - def check(self, spec, name): + def check_basic(self, spec, name, ispkg=False): self.assertEqual(spec.name, name) self.assertIs(spec.loader, self.machinery.FrozenImporter) self.assertEqual(spec.origin, 'frozen') self.assertFalse(spec.has_location) + if ispkg: + self.assertIsNotNone(spec.submodule_search_locations) + else: + self.assertIsNone(spec.submodule_search_locations) + self.assertIsNotNone(spec.loader_state) + + def check_search_location(self, spec, source=None): + # Frozen packages do not have any path entries. + # (See https://bugs.python.org/issue21736.) + expected = [] + self.assertListEqual(spec.submodule_search_locations, expected) + + def check_data(self, spec, source=None, ispkg=None): + with import_helper.frozen_modules(): + expected = _imp.get_frozen_object(spec.name) + data = spec.loader_state + # We can't compare the marshaled data directly because + # marshal.dumps() would mark "expected" as a ref, which slightly + # changes the output. (See https://bugs.python.org/issue34093.) + code = marshal.loads(data) + self.assertEqual(code, expected) def test_module(self): - names = [ - '__hello__', - '__hello_alias__', - '__hello_only__', - '__phello__.__init__', - '__phello__.spam', - '__phello__.ham.__init__', - '__phello__.ham.eggs', - ] - for name in names: + modules = { + '__hello__': None, + '__phello__.__init__': None, + '__phello__.spam': None, + '__phello__.ham.__init__': None, + '__phello__.ham.eggs': None, + '__hello_alias__': '__hello__', + } + for name, source in modules.items(): with self.subTest(name): spec = self.find(name) - self.check(spec, name) - self.assertEqual(spec.submodule_search_locations, None) + self.check_basic(spec, name) + self.check_data(spec, source) def test_package(self): - names = [ - '__phello__', - '__phello__.ham', - '__phello_alias__', - ] - for name in names: + modules = { + '__phello__': None, + '__phello__.ham': None, + '__phello_alias__': '__hello__', + } + for name, source in modules.items(): with self.subTest(name): spec = self.find(name) - self.check(spec, name) - self.assertEqual(spec.submodule_search_locations, []) + self.check_basic(spec, name, ispkg=True) + self.check_search_location(spec, source) + self.check_data(spec, source, ispkg=True) + + def test_frozen_only(self): + name = '__hello_only__' + source = os.path.join(REPO_ROOT, 'Tools', 'freeze', 'flag.py') + spec = self.find(name) + self.check_basic(spec, name) + self.check_data(spec, source) # These are covered by test_module() and test_package(). test_module_in_package = None diff --git a/Lib/test/test_importlib/frozen/test_loader.py b/Lib/test/test_importlib/frozen/test_loader.py index cfa5e5bb31bea..992dcef05bcb1 100644 --- a/Lib/test/test_importlib/frozen/test_loader.py +++ b/Lib/test/test_importlib/frozen/test_loader.py @@ -4,7 +4,9 @@ machinery = util.import_importlib('importlib.machinery') from test.support import captured_stdout, import_helper +import _imp import contextlib +import marshal import types import unittest import warnings @@ -33,11 +35,14 @@ class ExecModuleTests(abc.LoaderTests): def exec_module(self, name): with import_helper.frozen_modules(): is_package = self.machinery.FrozenImporter.is_package(name) + code = _imp.get_frozen_object(name) + data = marshal.dumps(code) spec = self.machinery.ModuleSpec( name, self.machinery.FrozenImporter, origin='frozen', is_package=is_package, + loader_state=data, ) module = types.ModuleType(name) module.__spec__ = spec @@ -61,6 +66,7 @@ def test_module(self): self.assertEqual(getattr(module, attr), value) self.assertEqual(output, 'Hello world!\n') self.assertTrue(hasattr(module, '__spec__')) + self.assertIsNone(module.__spec__.loader_state) def test_package(self): name = '__phello__' @@ -73,6 +79,7 @@ def test_package(self): name=name, attr=attr, given=attr_value, expected=value)) self.assertEqual(output, 'Hello world!\n') + self.assertIsNone(module.__spec__.loader_state) def test_lacking_parent(self): name = '__phello__.spam' diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-29-12-02-39.bpo-45324.BTQElX.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-29-12-02-39.bpo-45324.BTQElX.rst new file mode 100644 index 0000000000000..7b16847bb7339 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-29-12-02-39.bpo-45324.BTQElX.rst @@ -0,0 +1,3 @@ +In FrozenImporter.find_spec(), we now preserve the information needed in +exec_module() to load the module. This change mostly impacts internal +details, rather than changing the importer's behavior. diff --git a/Python/clinic/import.c.h b/Python/clinic/import.c.h index 438a348fa097f..09738834195c7 100644 --- a/Python/clinic/import.c.h +++ b/Python/clinic/import.c.h @@ -169,33 +169,80 @@ _imp_init_frozen(PyObject *module, PyObject *arg) return return_value; } -PyDoc_STRVAR(_imp_get_frozen_object__doc__, -"get_frozen_object($module, name, /)\n" +PyDoc_STRVAR(_imp_find_frozen__doc__, +"find_frozen($module, name, /)\n" "--\n" "\n" -"Create a code object for a frozen module."); +"Return info about the corresponding frozen module (if there is one) or None.\n" +"\n" +"The returned info (a 2-tuple):\n" +"\n" +" * data the raw marshalled bytes\n" +" * is_package whether or not it is a package"); -#define _IMP_GET_FROZEN_OBJECT_METHODDEF \ - {"get_frozen_object", (PyCFunction)_imp_get_frozen_object, METH_O, _imp_get_frozen_object__doc__}, +#define _IMP_FIND_FROZEN_METHODDEF \ + {"find_frozen", (PyCFunction)_imp_find_frozen, METH_O, _imp_find_frozen__doc__}, static PyObject * -_imp_get_frozen_object_impl(PyObject *module, PyObject *name); +_imp_find_frozen_impl(PyObject *module, PyObject *name); static PyObject * -_imp_get_frozen_object(PyObject *module, PyObject *arg) +_imp_find_frozen(PyObject *module, PyObject *arg) { PyObject *return_value = NULL; PyObject *name; if (!PyUnicode_Check(arg)) { - _PyArg_BadArgument("get_frozen_object", "argument", "str", arg); + _PyArg_BadArgument("find_frozen", "argument", "str", arg); goto exit; } if (PyUnicode_READY(arg) == -1) { goto exit; } name = arg; - return_value = _imp_get_frozen_object_impl(module, name); + return_value = _imp_find_frozen_impl(module, name); + +exit: + return return_value; +} + +PyDoc_STRVAR(_imp_get_frozen_object__doc__, +"get_frozen_object($module, name, data=None, /)\n" +"--\n" +"\n" +"Create a code object for a frozen module."); + +#define _IMP_GET_FROZEN_OBJECT_METHODDEF \ + {"get_frozen_object", (PyCFunction)(void(*)(void))_imp_get_frozen_object, METH_FASTCALL, _imp_get_frozen_object__doc__}, + +static PyObject * +_imp_get_frozen_object_impl(PyObject *module, PyObject *name, + PyObject *dataobj); + +static PyObject * +_imp_get_frozen_object(PyObject *module, PyObject *const *args, Py_ssize_t nargs) +{ + PyObject *return_value = NULL; + PyObject *name; + PyObject *dataobj = Py_None; + + if (!_PyArg_CheckPositional("get_frozen_object", nargs, 1, 2)) { + goto exit; + } + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("get_frozen_object", "argument 1", "str", args[0]); + goto exit; + } + if (PyUnicode_READY(args[0]) == -1) { + goto exit; + } + name = args[0]; + if (nargs < 2) { + goto skip_optional; + } + dataobj = args[1]; +skip_optional: + return_value = _imp_get_frozen_object_impl(module, name, dataobj); exit: return return_value; @@ -498,4 +545,4 @@ _imp_source_hash(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyOb #ifndef _IMP_EXEC_DYNAMIC_METHODDEF #define _IMP_EXEC_DYNAMIC_METHODDEF #endif /* !defined(_IMP_EXEC_DYNAMIC_METHODDEF) */ -/*[clinic end generated code: output=96038c277119d6e3 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=a31e1c00653359ff input=a9049054013a1b77]*/ diff --git a/Python/import.c b/Python/import.c index d7f126784192b..22cefdf08b48f 100644 --- a/Python/import.c +++ b/Python/import.c @@ -887,10 +887,6 @@ _imp__fix_co_filename_impl(PyObject *module, PyCodeObject *code, } -/* Forward */ -static const struct _frozen * find_frozen(PyObject *); - - /* Helper to test for built-in module */ static int @@ -1119,75 +1115,125 @@ list_frozen_module_names() return names; } -static const struct _frozen * -find_frozen(PyObject *modname) +typedef enum { + FROZEN_OKAY, + FROZEN_BAD_NAME, // The given module name wasn't valid. + FROZEN_NOT_FOUND, // It wasn't in PyImport_FrozenModules. + FROZEN_DISABLED, // -X frozen_modules=off (and not essential) + FROZEN_EXCLUDED, // The PyImport_FrozenModules entry has NULL "code". + FROZEN_INVALID, // The PyImport_FrozenModules entry is bogus. +} frozen_status; + +static inline void +set_frozen_error(frozen_status status, PyObject *modname) +{ + const char *err = NULL; + switch (status) { + case FROZEN_BAD_NAME: + case FROZEN_NOT_FOUND: + case FROZEN_DISABLED: + err = "No such frozen object named %R"; + break; + case FROZEN_EXCLUDED: + err = "Excluded frozen object named %R"; + break; + case FROZEN_INVALID: + err = "Frozen object named %R is invalid"; + break; + case FROZEN_OKAY: + // There was no error. + break; + default: + Py_UNREACHABLE(); + } + if (err != NULL) { + PyObject *msg = PyUnicode_FromFormat(err, modname); + if (msg == NULL) { + PyErr_Clear(); + } + PyErr_SetImportError(msg, modname, NULL); + Py_XDECREF(msg); + } +} + +struct frozen_info { + PyObject *nameobj; + const char *data; + Py_ssize_t size; + bool is_package; +}; + +static frozen_status +find_frozen(PyObject *nameobj, struct frozen_info *info) { - if (modname == NULL) { - return NULL; + if (info != NULL) { + info->nameobj = NULL; + info->data = NULL; + info->size = 0; + info->is_package = false; + } + + if (nameobj == NULL || nameobj == Py_None) { + return FROZEN_BAD_NAME; } - const char *name = PyUnicode_AsUTF8(modname); + const char *name = PyUnicode_AsUTF8(nameobj); if (name == NULL) { + // Note that this function previously used + // _PyUnicode_EqualToASCIIString(). We clear the error here + // (instead of propagating it) to match the earlier behavior + // more closely. PyErr_Clear(); - return NULL; + return FROZEN_BAD_NAME; } + if (!use_frozen() && !is_essential_frozen_module(name)) { - return NULL; + return FROZEN_DISABLED; } + const struct _frozen *p; for (p = PyImport_FrozenModules; ; p++) { if (p->name == NULL) { - return NULL; + // We hit the end-of-list sentinel value. + return FROZEN_NOT_FOUND; } if (strcmp(name, p->name) == 0) { break; } } - return p; -} - -static PyObject * -get_frozen_object(PyObject *name) -{ - const struct _frozen *p = find_frozen(name); - int size; - - if (p == NULL) { - PyErr_Format(PyExc_ImportError, - "No such frozen object named %R", - name); - return NULL; + if (info != NULL) { + info->nameobj = nameobj; // borrowed + info->data = (const char *)p->code; + info->size = p->size < 0 ? -(p->size) : p->size; + info->is_package = p->size < 0 ? true : false; } + if (p->code == NULL) { - PyErr_Format(PyExc_ImportError, - "Excluded frozen object named %R", - name); - return NULL; + /* It is frozen but marked as un-importable. */ + return FROZEN_EXCLUDED; } - size = p->size; - if (size < 0) - size = -size; - return PyMarshal_ReadObjectFromString((const char *)p->code, size); + if (p->code[0] == '\0' || p->size == 0) { + return FROZEN_INVALID; + } + return FROZEN_OKAY; } static PyObject * -is_frozen_package(PyObject *name) +unmarshal_frozen_code(struct frozen_info *info) { - const struct _frozen *p = find_frozen(name); - int size; - - if (p == NULL) { - PyErr_Format(PyExc_ImportError, - "No such frozen object named %R", - name); + PyObject *co = PyMarshal_ReadObjectFromString(info->data, info->size); + if (co == NULL) { + set_frozen_error(FROZEN_INVALID, info->nameobj); return NULL; } - - size = p->size; - - if (size < 0) - Py_RETURN_TRUE; - else - Py_RETURN_FALSE; + if (!PyCode_Check(co)) { + // We stick with TypeError for backward compatibility. + PyErr_Format(PyExc_TypeError, + "frozen object %R is not a code object", + info->nameobj); + Py_DECREF(co); + return NULL; + } + return co; } @@ -1200,35 +1246,25 @@ int PyImport_ImportFrozenModuleObject(PyObject *name) { PyThreadState *tstate = _PyThreadState_GET(); - const struct _frozen *p; PyObject *co, *m, *d; - int ispackage; - int size; - - p = find_frozen(name); - if (p == NULL) + struct frozen_info info; + frozen_status status = find_frozen(name, &info); + if (status == FROZEN_NOT_FOUND || status == FROZEN_DISABLED) { return 0; - if (p->code == NULL) { - _PyErr_Format(tstate, PyExc_ImportError, - "Excluded frozen object named %R", - name); + } + else if (status == FROZEN_BAD_NAME) { + return 0; + } + else if (status != FROZEN_OKAY) { + set_frozen_error(status, name); return -1; } - size = p->size; - ispackage = (size < 0); - if (ispackage) - size = -size; - co = PyMarshal_ReadObjectFromString((const char *)p->code, size); - if (co == NULL) + co = unmarshal_frozen_code(&info); + if (co == NULL) { return -1; - if (!PyCode_Check(co)) { - _PyErr_Format(tstate, PyExc_TypeError, - "frozen object %R is not a code object", - name); - goto err_return; } - if (ispackage) { + if (info.is_package) { /* Set __path__ to the empty list */ PyObject *l; int err; @@ -1966,20 +2002,83 @@ _imp_init_frozen_impl(PyObject *module, PyObject *name) return import_add_module(tstate, name); } +/*[clinic input] +_imp.find_frozen + + name: unicode + / + +Return info about the corresponding frozen module (if there is one) or None. + +The returned info (a 2-tuple): + + * data the raw marshalled bytes + * is_package whether or not it is a package +[clinic start generated code]*/ + +static PyObject * +_imp_find_frozen_impl(PyObject *module, PyObject *name) +/*[clinic end generated code: output=3fd17da90d417e4e input=4e52b3ac95f6d7ab]*/ +{ + struct frozen_info info; + frozen_status status = find_frozen(name, &info); + if (status == FROZEN_NOT_FOUND || status == FROZEN_DISABLED) { + Py_RETURN_NONE; + } + else if (status == FROZEN_BAD_NAME) { + Py_RETURN_NONE; + } + else if (status != FROZEN_OKAY) { + set_frozen_error(status, name); + return NULL; + } + PyObject *data = PyBytes_FromStringAndSize(info.data, info.size); + if (data == NULL) { + return NULL; + } + PyObject *result = PyTuple_Pack(2, data, + info.is_package ? Py_True : Py_False); + Py_DECREF(data); + return result; +} + /*[clinic input] _imp.get_frozen_object name: unicode + data as dataobj: object = None / Create a code object for a frozen module. [clinic start generated code]*/ static PyObject * -_imp_get_frozen_object_impl(PyObject *module, PyObject *name) -/*[clinic end generated code: output=2568cc5b7aa0da63 input=ed689bc05358fdbd]*/ -{ - return get_frozen_object(name); +_imp_get_frozen_object_impl(PyObject *module, PyObject *name, + PyObject *dataobj) +/*[clinic end generated code: output=54368a673a35e745 input=034bdb88f6460b7b]*/ +{ + struct frozen_info info; + if (PyBytes_Check(dataobj)) { + info.nameobj = name; + info.data = PyBytes_AS_STRING(dataobj); + info.size = PyBytes_Size(dataobj); + if (info.size == 0) { + set_frozen_error(FROZEN_INVALID, name); + return NULL; + } + } + else if (dataobj != Py_None) { + _PyArg_BadArgument("get_frozen_object", "argument 2", "bytes", dataobj); + return NULL; + } + else { + frozen_status status = find_frozen(name, &info); + if (status != FROZEN_OKAY) { + set_frozen_error(status, name); + return NULL; + } + } + return unmarshal_frozen_code(&info); } /*[clinic input] @@ -1995,7 +2094,13 @@ static PyObject * _imp_is_frozen_package_impl(PyObject *module, PyObject *name) /*[clinic end generated code: output=e70cbdb45784a1c9 input=81b6cdecd080fbb8]*/ { - return is_frozen_package(name); + struct frozen_info info; + frozen_status status = find_frozen(name, &info); + if (status != FROZEN_OKAY && status != FROZEN_EXCLUDED) { + set_frozen_error(status, name); + return NULL; + } + return PyBool_FromLong(info.is_package); } /*[clinic input] @@ -2027,10 +2132,12 @@ static PyObject * _imp_is_frozen_impl(PyObject *module, PyObject *name) /*[clinic end generated code: output=01f408f5ec0f2577 input=7301dbca1897d66b]*/ { - const struct _frozen *p; - - p = find_frozen(name); - return PyBool_FromLong((long) (p == NULL ? 0 : p->size)); + struct frozen_info info; + frozen_status status = find_frozen(name, &info); + if (status != FROZEN_OKAY) { + Py_RETURN_FALSE; + } + Py_RETURN_TRUE; } /*[clinic input] @@ -2221,6 +2328,7 @@ static PyMethodDef imp_methods[] = { _IMP_LOCK_HELD_METHODDEF _IMP_ACQUIRE_LOCK_METHODDEF _IMP_RELEASE_LOCK_METHODDEF + _IMP_FIND_FROZEN_METHODDEF _IMP_GET_FROZEN_OBJECT_METHODDEF _IMP_IS_FROZEN_PACKAGE_METHODDEF _IMP_CREATE_BUILTIN_METHODDEF From webhook-mailer at python.org Tue Oct 5 12:35:09 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 16:35:09 -0000 Subject: [Python-checkins] bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) (GH-28738) Message-ID: https://github.com/python/cpython/commit/d0d29655ffc43d426ad68542d8de8304f7f1346a commit: d0d29655ffc43d426ad68542d8de8304f7f1346a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-05T18:34:59+02:00 summary: bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) (GH-28738) Automerge-Triggered-By: GH:encukou (cherry picked from commit b9bb74871b27d9226df2dd3fce9d42bda8b43c2b) Co-authored-by: Hai Shi files: A Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst M Lib/test/test_capi.py M Modules/_testmultiphase.c M Python/import.c diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 5f78c337028a9..db26b9b5ddedb 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -762,6 +762,37 @@ def test_mutate_exception(self): self.assertFalse(hasattr(binascii.Error, "foobar")) + def test_module_state_shared_in_global(self): + """ + bpo-44050: Extension module state should be shared between interpreters + when it doesn't support sub-interpreters. + """ + r, w = os.pipe() + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + + script = textwrap.dedent(f""" + import importlib.machinery + import importlib.util + import os + + fullname = '_test_module_state_shared' + origin = importlib.util.find_spec('_testmultiphase').origin + loader = importlib.machinery.ExtensionFileLoader(fullname, origin) + spec = importlib.util.spec_from_loader(fullname, loader) + module = importlib.util.module_from_spec(spec) + attr_id = str(id(module.Error)).encode() + + os.write({w}, attr_id) + """) + exec(script) + main_attr_id = os.read(r, 100) + + ret = support.run_in_subinterp(script) + self.assertEqual(ret, 0) + subinterp_attr_id = os.read(r, 100) + self.assertEqual(main_attr_id, subinterp_attr_id) + class TestThreadState(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst new file mode 100644 index 0000000000000..d6eed9f1bcfe9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst @@ -0,0 +1,3 @@ +Extensions that indicate they use global state (by setting ``m_size`` to -1) +can again be used in multiple interpreters. This reverts to behavior of +Python 3.8. diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index ad60f32f7e7a6..e0ed77d265cdc 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -844,6 +844,28 @@ PyInit__testmultiphase_meth_state_access(PyObject *spec) return PyModuleDef_Init(&def_meth_state_access); } +static PyModuleDef def_module_state_shared = { + PyModuleDef_HEAD_INIT, + .m_name = "_test_module_state_shared", + .m_doc = PyDoc_STR("Regression Test module for single-phase init."), + .m_size = -1, +}; + +PyMODINIT_FUNC +PyInit__test_module_state_shared(PyObject *spec) +{ + PyObject *module = PyModule_Create(&def_module_state_shared); + if (module == NULL) { + return NULL; + } + + if (PyModule_AddObjectRef(module, "Error", PyExc_Exception) < 0) { + Py_DECREF(module); + return NULL; + } + return module; +} + /*** Helper for imp test ***/ diff --git a/Python/import.c b/Python/import.c index f2b30afe3f228..50f4956e5a9d6 100644 --- a/Python/import.c +++ b/Python/import.c @@ -441,7 +441,9 @@ _PyImport_FixupExtensionObject(PyObject *mod, PyObject *name, return -1; } - if (_Py_IsMainInterpreter(tstate->interp)) { + // bpo-44050: Extensions and def->m_base.m_copy can be updated + // when the extension module doesn't support sub-interpreters. + if (_Py_IsMainInterpreter(tstate->interp) || def->m_size == -1) { if (def->m_size == -1) { if (def->m_base.m_copy) { /* Somebody already imported the module, From webhook-mailer at python.org Tue Oct 5 12:37:22 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Tue, 05 Oct 2021 16:37:22 -0000 Subject: [Python-checkins] Rearrage the finder tests. (gh-28740) Message-ID: https://github.com/python/cpython/commit/69f6dabb9c469f308650b820287e4cb0eac7e655 commit: 69f6dabb9c469f308650b820287e4cb0eac7e655 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-05T10:37:14-06:00 summary: Rearrage the finder tests. (gh-28740) This makes the tests a bit cleaner and makes adding more tests a little simpler. https://bugs.python.org/issue45324 files: M Lib/test/test_importlib/frozen/test_finder.py diff --git a/Lib/test/test_importlib/frozen/test_finder.py b/Lib/test/test_importlib/frozen/test_finder.py index 23d1bf7fb7773..0b15aeb598dd9 100644 --- a/Lib/test/test_importlib/frozen/test_finder.py +++ b/Lib/test/test_importlib/frozen/test_finder.py @@ -32,13 +32,7 @@ def check_basic(self, spec, name, ispkg=False): self.assertIsNone(spec.submodule_search_locations) self.assertIsNotNone(spec.loader_state) - def check_search_location(self, spec, source=None): - # Frozen packages do not have any path entries. - # (See https://bugs.python.org/issue21736.) - expected = [] - self.assertListEqual(spec.submodule_search_locations, expected) - - def check_data(self, spec, source=None, ispkg=None): + def check_data(self, spec): with import_helper.frozen_modules(): expected = _imp.get_frozen_object(spec.name) data = spec.loader_state @@ -48,40 +42,72 @@ def check_data(self, spec, source=None, ispkg=None): code = marshal.loads(data) self.assertEqual(code, expected) + def check_search_locations(self, spec): + # Frozen packages do not have any path entries. + # (See https://bugs.python.org/issue21736.) + expected = [] + self.assertListEqual(spec.submodule_search_locations, expected) + def test_module(self): + modules = [ + '__hello__', + '__phello__.spam', + '__phello__.ham.eggs', + ] + for name in modules: + with self.subTest(f'{name} -> {name}'): + spec = self.find(name) + self.check_basic(spec, name) + self.check_data(spec) modules = { - '__hello__': None, - '__phello__.__init__': None, - '__phello__.spam': None, - '__phello__.ham.__init__': None, - '__phello__.ham.eggs': None, '__hello_alias__': '__hello__', - } - for name, source in modules.items(): - with self.subTest(name): + '_frozen_importlib': 'importlib._bootstrap', + } + for name, origname in modules.items(): + with self.subTest(f'{name} -> {origname}'): + spec = self.find(name) + self.check_basic(spec, name) + self.check_data(spec) + modules = [ + '__phello__.__init__', + '__phello__.ham.__init__', + ] + for name in modules: + origname = name.rpartition('.')[0] + with self.subTest(f'{name} -> {origname}'): spec = self.find(name) self.check_basic(spec, name) - self.check_data(spec, source) + self.check_data(spec) + modules = { + '__hello_only__': ('Tools', 'freeze', 'flag.py'), + } + for name, path in modules.items(): + filename = os.path.join(REPO_ROOT, *path) + with self.subTest(f'{name} -> {filename}'): + spec = self.find(name) + self.check_basic(spec, name) + self.check_data(spec) def test_package(self): - modules = { - '__phello__': None, - '__phello__.ham': None, + packages = [ + '__phello__', + '__phello__.ham', + ] + for name in packages: + with self.subTest(f'{name} -> {name}'): + spec = self.find(name) + self.check_basic(spec, name, ispkg=True) + self.check_data(spec) + self.check_search_locations(spec) + packages = { '__phello_alias__': '__hello__', } - for name, source in modules.items(): - with self.subTest(name): + for name, origname in packages.items(): + with self.subTest(f'{name} -> {origname}'): spec = self.find(name) self.check_basic(spec, name, ispkg=True) - self.check_search_location(spec, source) - self.check_data(spec, source, ispkg=True) - - def test_frozen_only(self): - name = '__hello_only__' - source = os.path.join(REPO_ROOT, 'Tools', 'freeze', 'flag.py') - spec = self.find(name) - self.check_basic(spec, name) - self.check_data(spec, source) + self.check_data(spec) + self.check_search_locations(spec) # These are covered by test_module() and test_package(). test_module_in_package = None From webhook-mailer at python.org Tue Oct 5 12:42:10 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 16:42:10 -0000 Subject: [Python-checkins] Update macOS installer ReadMe for 3.9.8. (GH-28701) Message-ID: https://github.com/python/cpython/commit/f4a7ce70f9e61b8594f0979ae0e5f39868d4ceb2 commit: f4a7ce70f9e61b8594f0979ae0e5f39868d4ceb2 branch: 3.9 author: Ned Deily committer: ambv date: 2021-10-05T18:42:02+02:00 summary: Update macOS installer ReadMe for 3.9.8. (GH-28701) The universal2 installer variant is now the default download from python.org and the legacy Intel-64 variant is now deprecated. files: M Mac/BuildScript/resources/ReadMe.rtf diff --git a/Mac/BuildScript/resources/ReadMe.rtf b/Mac/BuildScript/resources/ReadMe.rtf index 4eb18b7bdaf44..6cc11d752540e 100644 --- a/Mac/BuildScript/resources/ReadMe.rtf +++ b/Mac/BuildScript/resources/ReadMe.rtf @@ -60,16 +60,20 @@ Due to new security checks on macOS 10.15 Catalina, when launching IDLE macOS ma \f0\b0 button to proceed.\ \ -\f1\b \ul macOS 11 (Big Sur) and Apple Silicon Mac support [updated in 3.9.5]\ +\f1\b \ul macOS 11 (Big Sur) and Apple Silicon Mac support [updated in 3.9.8]\ \f0\b0 \ulnone \ -As of 2020-11, macOS 11.0 (Big Sur) is the latest release of macOS and one of its major features is the support of new Apple Silicon Macs that are based on the ARM64 CPU architecture specification rather than the Intel 64 (x86_64) architecture used previously. There are other changes in Big Sur that affect Python operation regardless of CPU architecture. As of 3.9.1, Python binaries from python.org fully support Big Sur. \ + +\f1\b NEW as of 3.9.8 +\f0\b0 : The +\f4 universal2 +\f0 installer variant is now the default download from python.org and the legacy Intel-only variant is deprecated.\ \ -python.org binaries for macOS have been provided via a downloadable installer that supports the Intel 64 architecture on macOS 10.9 and newer. This installer variant has been the default download for 3.9.1; it will install and run on all Macs that run macOS 10.9 or later, including 11.0 (Big Sur). This variant +Prior to the 3.9.1 release, python.org binaries for macOS have been provided via a downloadable installer that supports the Intel 64 architecture on macOS 10.9 and newer. It will install and run on all Macs that run macOS 10.9 or later, including 11.0 (Big Sur). This variant \f2\i should \f0\i0 run transparently on new Apple Silicon Macs using Apple's Rosetta 2 emulation. \ \ -Beginning with 3.9.1, we provide a new +Beginning with 3.9.1, we also provide a new \f4 universal2 \f0 installer variant that provides universal binaries for both \f4 ARM64 @@ -79,23 +83,23 @@ Beginning with 3.9.1, we provide a new \ On Apple Silicon Macs with the \f4 universal2 -\f0 variant, it is possible to run Python either with native ARM64 code or under Intel 64 emulation using Rosetta 2. This option might be useful for testing or if binary wheels are not yet available with native ARM64 binaries. To easily force Python to run in emulation mode, invoke it from a command line shell with the +\f0 variant installed, it is possible to run Python either with native ARM64 code or under Intel 64 emulation using Rosetta 2. This option might be useful for testing or if binary wheels are not yet available with native ARM64 binaries. To easily force Python to run in emulation mode on an Apple Silicon Mac, invoke it from a command line shell with the \f4 python3-intel64 \f0 (or -\f4 python3.10-intel64 +\f4 python3.9-intel64 \f0 ) command (new with 3.9.5) instead of just -\f4 python3 +\f4 python3 (or python3.9) \f0 . \ \ Binary wheels built for use with the legacy 10.9 variant \f2\i should \f0\i0 also work with the new variant on Intel-based Macs or when running under Rosetta2 emulation on Apple Silicon Macs. \ \ -As of the 3.9.5 release, we encourage you to use the +We encourage you to use the \f4 universal2 -\f0 variant whenever possible. The legacy 10.9+ Intel-only variant will not be provided for Python 3.10 and the +\f0 variant whenever possible. The legacy 10.9+ Intel-only variant will not be provided for Python 3.10 and, as of the 3.9.8 release, the \f4 universal2 -\f0 variant will become the default download for future 3.9.x releases. You may need to upgrade third-party components, like +\f0 variant is now the default download. You may need to upgrade third-party components, like \f4 pip \f0 , to later versions once they are released. You may experience differences in behavior in \f4 IDLE From webhook-mailer at python.org Tue Oct 5 12:53:43 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 16:53:43 -0000 Subject: [Python-checkins] [doc] Fix gethostbyname_ex description (GH-28700) Message-ID: https://github.com/python/cpython/commit/4103280b83e1419bef535a42813d6dbe83bfe880 commit: 4103280b83e1419bef535a42813d6dbe83bfe880 branch: main author: Andre Delfino committer: ambv date: 2021-10-05T18:53:35+02:00 summary: [doc] Fix gethostbyname_ex description (GH-28700) It seems part of `gethostbyname_ex` doc was copied from `gethostbyaddr`. The latter has an `ip_address` parameter whereas the former doesn't. files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 397ba54733ca8..27ad5c7fd1d91 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -827,8 +827,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: gethostbyname_ex(hostname) Translate a host name to IPv4 address format, extended interface. Return a - triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the primary - host name responding to the given *ip_address*, *aliaslist* is a (possibly + triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the host's + primary host name, *aliaslist* is a (possibly empty) list of alternative host names for the same address, and *ipaddrlist* is a list of IPv4 addresses for the same interface on the same host (often but not always a single address). :func:`gethostbyname_ex` does not support IPv6 name From webhook-mailer at python.org Tue Oct 5 13:08:52 2021 From: webhook-mailer at python.org (pablogsal) Date: Tue, 05 Oct 2021 17:08:52 -0000 Subject: [Python-checkins] Python 3.11.0a1 Message-ID: https://github.com/python/cpython/commit/7c12e4835ebe52287acd200a2e76b533413b15d0 commit: 7c12e4835ebe52287acd200a2e76b533413b15d0 branch: main author: Pablo Galindo committer: pablogsal date: 2021-10-05T13:44:05+01:00 summary: Python 3.11.0a1 files: A Misc/NEWS.d/3.11.0a1.rst D Misc/NEWS.d/next/Build/2021-05-24-03-31-17.bpo-41282.L8nP44.rst D Misc/NEWS.d/next/Build/2021-06-10-18-08-44.bpo-44381.Xpc1iX.rst D Misc/NEWS.d/next/Build/2021-06-19-11-50-03.bpo-43298.9ircMb.rst D Misc/NEWS.d/next/Build/2021-06-30-02-32-46.bpo-44535.M9iN4-.rst D Misc/NEWS.d/next/Build/2021-07-19-01-09-56.bpo-44340.JNeOf4.rst D Misc/NEWS.d/next/Build/2021-08-26-13-10-46.bpo-45019.e0mo49.rst D Misc/NEWS.d/next/Build/2021-09-09-16-45-26.bpo-45067.mFmY92.rst D Misc/NEWS.d/next/Build/2021-09-11-06-05-23.bpo-45163.q7xT93.rst D Misc/NEWS.d/next/Build/2021-09-14-00-47-57.bpo-45188.MNbo_T.rst D Misc/NEWS.d/next/Build/2021-09-14-10-07-23.bpo-45020._VGGPv.rst D Misc/NEWS.d/next/Build/2021-09-16-18-00-43.bpo-45220.TgbkvW.rst D Misc/NEWS.d/next/C API/2020-12-23-01-28-50.bpo-42035.S9eUm0.rst D Misc/NEWS.d/next/C API/2021-05-04-17-43-39.bpo-44029.ayX4PR.rst D Misc/NEWS.d/next/C API/2021-05-05-19-04-50.bpo-43795.9Ojj73.rst D Misc/NEWS.d/next/C API/2021-05-10-14-34-22.bpo-44094.HayXZO.rst D Misc/NEWS.d/next/C API/2021-05-12-12-24-45.bpo-44113.DcgOqE.rst D Misc/NEWS.d/next/C API/2021-05-19-15-09-47.bpo-43795.WAHRxt.rst D Misc/NEWS.d/next/C API/2021-05-31-11-31-13.bpo-44263.8mIOfV.rst D Misc/NEWS.d/next/C API/2021-06-03-00-59-48.bpo-39573.-elHTJ.rst D Misc/NEWS.d/next/C API/2021-06-10-15-22-31.bpo-44378.jGYakF.rst D Misc/NEWS.d/next/C API/2021-06-15-16-28-18.bpo-43795.fy0AXK.rst D Misc/NEWS.d/next/C API/2021-06-22-17-00-06.bpo-40939.CGB0I5.rst D Misc/NEWS.d/next/C API/2021-06-23-10-31-45.bpo-39947.je_HMo.rst D Misc/NEWS.d/next/C API/2021-06-23-12-12-04.bpo-44441.3p14JB.rst D Misc/NEWS.d/next/C API/2021-06-28-23-44-47.bpo-44530.qij7YC.rst D Misc/NEWS.d/next/C API/2021-07-20-16-21-06.bpo-42747.rRxjUY.rst D Misc/NEWS.d/next/C API/2021-07-27-17-29-12.bpo-44751.4qmbDG.rst D Misc/NEWS.d/next/C API/2021-07-29-16-04-28.bpo-41103.hiKKcF.rst D Misc/NEWS.d/next/C API/2021-08-02-20-49-36.bpo-42035.HTBcZt.rst D Misc/NEWS.d/next/C API/2021-08-31-15-21-36.bpo-45061.ZH0HVe.rst D Misc/NEWS.d/next/C API/2021-09-03-15-53-43.bpo-45094.tinXwL.rst D Misc/NEWS.d/next/C API/2021-09-16-18-05-20.bpo-45116.WxXewl.rst D Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst D Misc/NEWS.d/next/C API/2021-09-30-03-14-35.bpo-41710.DDWJKx.rst D Misc/NEWS.d/next/Core and Builtins/2018-05-11-12-44-03.bpo-33346.ZgBkvB.rst D Misc/NEWS.d/next/Core and Builtins/2019-12-21-14-18-32.bpo-39091.dOexgQ.rst D Misc/NEWS.d/next/Core and Builtins/2020-06-02-13-21-14.bpo-11105.wceryW.rst D Misc/NEWS.d/next/Core and Builtins/2021-01-13-19-34-41.bpo-28146.AZBBkH.rst D Misc/NEWS.d/next/Core and Builtins/2021-03-22-17-50-30.bpo-17792._zssjS.rst D Misc/NEWS.d/next/Core and Builtins/2021-04-02-15-02-16.bpo-43693.l3Ureu.rst D Misc/NEWS.d/next/Core and Builtins/2021-04-17-16-08-00.bpo-43879.zkyJgh.rst D Misc/NEWS.d/next/Core and Builtins/2021-04-18-18-07-33.bpo-43833.oChkCi.rst D Misc/NEWS.d/next/Core and Builtins/2021-04-23-03-46-45.bpo-43918.nNDY3S.rst D Misc/NEWS.d/next/Core and Builtins/2021-04-30-15-48-36.bpo-40222.j3VxeX.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-04-01-01-04.bpo-43822.9VeCg0.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-08-17-18-37.bpo-43149.Kp5FxD.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-08-19-54-57.bpo-28307.7ysaVW.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-10-18-49-13.bpo-26110.EQzqqA.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-11-21-52-44.bpo-44110.VqbAsB.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-12-14-26-16.bpo-44114.p-WfAE.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-14-20-03-32.bpo-44032.OzT1ob.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-15-17-30-57.bpo-44143.7UTS6H.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-17-20-44-45.bpo-44156.8KSp9l.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-18-11-27-02.bpo-44168.mgB-rt.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-19-20-33-36.bpo-44180.mQVaAs.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-20-12-43-04.bpo-44187.3lk0L1.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-21-01-42-45.bpo-44184.9qOptC.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-21-20-53-49.bpo-43693.-NN3J_.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-21-21-16-03.bpo-44201.bGaSjt.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-25-18-20-10.bpo-44232.DMcCCf.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-26-19-10-47.bpo-43693.1KSG9u.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-27-17-34-29.bpo-43667.ND9jP3.rst D Misc/NEWS.d/next/Core and Builtins/2021-05-30-16-37-47.bpo-43413.vYFPPC.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-03-22-51-50.bpo-44305.66dVDG.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-06-00-29-14.bpo-44317.xPPhcZ.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-07-15-13-44.bpo-43693.c_zDeY.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-08-01-13-47.bpo-44335.GQTTkl.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-08-10-22-46.bpo-44337.RTjmIt.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-08-22-49-06.bpo-44349.xgEgeA.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-09-22-56-59.bpo-44368.vgT0Cx.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-10-10-06-18.bpo-44338.c4Myr4.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-10-16-10-39.bpo-44313.34RjI8.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-11-17-37-15.bpo-44376.zhM1UW.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-11-18-17-42.bpo-44396.Z9EKim.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-13-23-12-18.bpo-44409.eW4LS-.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-18-22-08-25.bpo-44456.L0Rhko.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-19-12-41-13.bpo-44297.F53vHj.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-20-10-53-21.bpo-12022.SW240M.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-21-11-19-54.bpo-44472.Vvm1yn.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-22-10-55-23.bpo-44486.wct-9X.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-22-19-08-19.bpo-44483.eq2f7T.rst D Misc/NEWS.d/next/Core and Builtins/2021-06-29-11-49-29.bpo-44523.67-ZIP.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-01-11-59-34.bpo-44490.xY80VR.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-02-22-54-41.bpo-44553.l9YqGg.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-03-00-20-39.bpo-43908.YHuV_s.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-04-17-41-47.bpo-41486.DiM24a.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-04-23-38-51.bpo-44562.QdeRQo.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-06-22-22-15.bpo-44490.BJxPbZ.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-07-16-05-35.bpo-43895.JFjR0-.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-08-12-18-56.bpo-44584.qKnSqV.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-09-12-08-17.bpo-44590.a2ntVX.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-12-04-06-57.bpo-41972.nDX5k_.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-13-17-47-32.bpo-42073.9wopiC.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-13-20-22-12.bpo-44606.S3Bv2w.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-13-23-19-41.bpo-44589.59OH8T.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-14-10-31-10.bpo-26280.cgpM4B.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-14-13-54-07.bpo-44635.7ZMAdB.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-16-01-01-11.bpo-44611.LcfHN-.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-36-12.bpo-44636.ZWebi8.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-59-13.bpo-44646.Yb6s05.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-16-20-25-37.bpo-44655.I3wRjL.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-16-21-35-14.bpo-44655.95I7M6.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-17-13-41-58.bpo-44662.q22kWR.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-17-14-20-59.bpo-44661.BQbXiH.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-17-21-04-04.bpo-44633.5-zKeI.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-19-20-49-06.bpo-44653.WcqGyI.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-21-15-26-56.bpo-44698.DA4_0o.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-23-01-52-13.bpo-44717.-vVmAh.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-23-15-17-01.bpo-44725.qcuKaa.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-26-15-27-03.bpo-44732.IxObt3.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-27-11-14-22.bpo-34013.SjLFe-.rst D Misc/NEWS.d/next/Core and Builtins/2021-07-31-12-12-57.bpo-44792.mOReTW.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-04-11-37-38.bpo-44821.67YHGI.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-42-03.bpo-44838.r_Lkj_.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-49-55.bpo-44826.zQsyK5.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-07-01-26-12.bpo-44856.9rk3li.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-07-21-39-19.bpo-25782.B22lMx.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-09-14-29-52.bpo-33930.--5LQ-.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-09-16-16-03.bpo-44872.OKRlhK.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-09-19-05-20.bpo-44874.oOcfU4.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-11-12-03-52.bpo-44878.nEhjLi.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-11-14-12-41.bpo-44878.pAbBfc.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-11-15-39-57.bpo-44885.i4noUO.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-11-16-46-27.bpo-44890.PwNg8N.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-11-20-45-02.bpo-44889.2T3nTn.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-12-14-00-57.bpo-44900.w2gpwy.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-14-20-13-21.bpo-44895.Ic9m90.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-15-10-39-06.bpo-44698.lITKNc.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-16-11-36-02.bpo-44914.6Lgrx3.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-16-23-16-17.bpo-44929.qpMEky.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-18-11-14-38.bpo-44945.CO3s77.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-18-19-09-28.bpo-44947.mcvGdS.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-19-14-43-24.bpo-44954.dLn3lg.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-22-12-28-50.bpo-24234.n3oTdx.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-23-10-36-55.bpo-24234.MGVUQi.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-23-19-55-08.bpo-44962.J00ftt.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-07-10.bpo-44963.5EET8y.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-17-32.bpo-45000.XjmyLl.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-26-18-44-03.bpo-45018.pu8H9L.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-31-11-09-52.bpo-45012.ueeOcx.rst D Misc/NEWS.d/next/Core and Builtins/2021-08-31-17-44-51.bpo-45020.ZPI_3L.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-01-16-55-43.bpo-45056.7AK2d9.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-01-19-21-48.bpo-34561.uMAVA-.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-01-23-55-49.bpo-45083.cLi9G3.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-02-01-28-01.bpo-37330.QDjM_l.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-03-12-35-17.bpo-41031.yPSJEs.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-03-16-18-10.bpo-1514420.2Lumpj.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-06-21-52-45.bpo-45123.8Eh9iI.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-07-00-21-04.bpo-44348.f8w_Td.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-07-17-10-16.bpo-45121.iG-Hsf.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-09-10-32-33.bpo-44219.WiYyjz.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-09-15-05-17.bpo-45155.JRw9TG.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-14-09-23-59.bpo-45167.CPSSoV.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-14-10-02-12.bpo-45190.ZFRgSj.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-20-10-02-12.bpo-24076.ZFgFSj.rst D Misc/NEWS.d/next/Core and Builtins/2021-09-21-22-27-25.bpo-45061.5IOUf0.rst D Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst D Misc/NEWS.d/next/Documentation/2018-05-19-15-59-29.bpo-33479.4cLlxo.rst D Misc/NEWS.d/next/Documentation/2020-01-30-05-18-48.bpo-39498.Nu3sFL.rst D Misc/NEWS.d/next/Documentation/2020-03-21-01-19-28.bpo-21760.CqofIc.rst D Misc/NEWS.d/next/Documentation/2020-08-21-22-59-37.bpo-41576.7a6CQR.rst D Misc/NEWS.d/next/Documentation/2020-08-24-13-35-04.bpo-41621.nqaw9G.rst D Misc/NEWS.d/next/Documentation/2020-09-03-13-37-19.bpo-41706._zXWOR.rst D Misc/NEWS.d/next/Documentation/2021-05-03-22-08-08.bpo-44025.gcB7iP.rst D Misc/NEWS.d/next/Documentation/2021-05-07-12-27-09.bpo-43558.UGhA8R.rst D Misc/NEWS.d/next/Documentation/2021-05-08-09-48-05.bpo-44072.fb2x5I.rst D Misc/NEWS.d/next/Documentation/2021-05-17-20-03-47.bpo-41963.eUz9_o.rst D Misc/NEWS.d/next/Documentation/2021-05-23-09-11-28.bpo-44195.1bqkOs.rst D Misc/NEWS.d/next/Documentation/2021-05-26-11-16-33.bpo-42392.oxRx6E.rst D Misc/NEWS.d/next/Documentation/2021-06-06-14-12-00.bpo-44322.K0PHfE.rst D Misc/NEWS.d/next/Documentation/2021-06-14-09-20-37.bpo-38291.VMYa_Q.rst D Misc/NEWS.d/next/Documentation/2021-06-16-18-09-49.bpo-44392.6RF1Sc.rst D Misc/NEWS.d/next/Documentation/2021-06-18-06-44-45.bpo-44453.3PIkj2.rst D Misc/NEWS.d/next/Documentation/2021-06-18-18-04-53.bpo-27752.NEByNk.rst D Misc/NEWS.d/next/Documentation/2021-06-21-15-46-32.bpo-13814.LDcslu.rst D Misc/NEWS.d/next/Documentation/2021-06-23-15-21-36.bpo-39452.o_I-6d.rst D Misc/NEWS.d/next/Documentation/2021-06-24-14-37-16.bpo-43066.Ti7ahX.rst D Misc/NEWS.d/next/Documentation/2021-06-26-17-41-06.bpo-40620.PAYDrB.rst D Misc/NEWS.d/next/Documentation/2021-06-28-12-13-48.bpo-38062.9Ehp9O.rst D Misc/NEWS.d/next/Documentation/2021-07-02-14-02-29.bpo-44544._5_aCz.rst D Misc/NEWS.d/next/Documentation/2021-07-03-18-25-17.bpo-44558.0pTknl.rst D Misc/NEWS.d/next/Documentation/2021-07-12-11-39-20.bpo-44613.DIXNzc.rst D Misc/NEWS.d/next/Documentation/2021-07-13-22-25-13.bpo-44631.qkGwe4.rst D Misc/NEWS.d/next/Documentation/2021-07-15-11-19-03.bpo-42958.gC5IHM.rst D Misc/NEWS.d/next/Documentation/2021-07-18-22-26-02.bpo-44651.SjT9iY.rst D Misc/NEWS.d/next/Documentation/2021-07-18-22-43-14.bpo-44561.T7HpWm.rst D Misc/NEWS.d/next/Documentation/2021-07-20-21-03-18.bpo-30511.eMFkRi.rst D Misc/NEWS.d/next/Documentation/2021-07-22-08-28-03.bpo-35183.p9BWTB.rst D Misc/NEWS.d/next/Documentation/2021-07-25-23-04-15.bpo-44693.JuCbNq.rst D Misc/NEWS.d/next/Documentation/2021-07-26-23-48-31.bpo-44740.zMFGMV.rst D Misc/NEWS.d/next/Documentation/2021-08-09-19-58-45.bpo-36700.WPNW5f.rst D Misc/NEWS.d/next/Documentation/2021-08-11-18-02-06.bpo-33479.rCe4c5.rst D Misc/NEWS.d/next/Documentation/2021-08-13-19-08-03.bpo-44903.aJuvQF.rst D Misc/NEWS.d/next/Documentation/2021-08-13-20-17-59.bpo-16580.MZ_iK9.rst D Misc/NEWS.d/next/Documentation/2021-08-19-15-53-08.bpo-44957.imqrh3.rst D Misc/NEWS.d/next/Documentation/2021-09-08-17-20-19.bpo-45024.dkNPNi.rst D Misc/NEWS.d/next/Documentation/2021-09-18-13-45-19.bpo-45216.o56nyt.rst D Misc/NEWS.d/next/IDLE/2021-05-05-09-45-24.bpo-44026.m2Z0zR.rst D Misc/NEWS.d/next/IDLE/2021-05-09-09-02-09.bpo-44010.TaLe9x.rst D Misc/NEWS.d/next/IDLE/2021-05-27-13-39-43.bpo-41611.liNQqj.rst D Misc/NEWS.d/next/IDLE/2021-05-27-18-22-46.bpo-41611.jOKpfc.rst D Misc/NEWS.d/next/IDLE/2021-06-08-03-04-51.bpo-40468.tUCGUb.rst D Misc/NEWS.d/next/IDLE/2021-06-10-00-50-02.bpo-33962.ikAUNg.rst D Misc/NEWS.d/next/IDLE/2021-06-11-17-43-39.bpo-40128.7vDN3U.rst D Misc/NEWS.d/next/IDLE/2021-09-15-03-20-06.bpo-45193.G61_GV.rst D Misc/NEWS.d/next/IDLE/2021-09-27-01-21-59.bpo-45296.9H8rdY.rst D Misc/NEWS.d/next/Library/2017-09-20-14-43-03.bpo-29298._78CSN.rst D Misc/NEWS.d/next/Library/2018-04-24-14-25-07.bpo-33349.Y_0LIr.rst D Misc/NEWS.d/next/Library/2019-02-26-09-31-59.bpo-26228.wyrHKc.rst D Misc/NEWS.d/next/Library/2019-05-08-15-14-32.bpo-16379.rN5JVe.rst D Misc/NEWS.d/next/Library/2019-06-03-23-53-25.bpo-27513.qITN7d.rst D Misc/NEWS.d/next/Library/2019-09-25-13-54-41.bpo-30256.wBkzox.rst D Misc/NEWS.d/next/Library/2019-10-08-14-08-59.bpo-38415.N1bUw6.rst D Misc/NEWS.d/next/Library/2019-11-12-18-59-33.bpo-38741.W7IYkq.rst D Misc/NEWS.d/next/Library/2020-01-16-13-54-28.bpo-39359.hzTu0h.rst D Misc/NEWS.d/next/Library/2020-01-16-23-41-16.bpo-38840.VzzYZz.rst D Misc/NEWS.d/next/Library/2020-01-25-12-58-20.bpo-37022.FUZI25.rst D Misc/NEWS.d/next/Library/2020-02-03-21-18-31.bpo-39549.l4a8uH.rst D Misc/NEWS.d/next/Library/2020-04-24-20-39-38.bpo-34990.3SmL9M.rst D Misc/NEWS.d/next/Library/2020-05-21-01-42-32.bpo-40563.fDn5bP.rst D Misc/NEWS.d/next/Library/2020-05-25-23-58-29.bpo-5846.O9BIfm.rst D Misc/NEWS.d/next/Library/2020-05-30-10-48-04.bpo-24391.ZCTnhX.rst D Misc/NEWS.d/next/Library/2020-07-01-17-42-41.bpo-41137.AnqbP-.rst D Misc/NEWS.d/next/Library/2020-07-13-23-46-59.bpo-32695.tTqqXe.rst D Misc/NEWS.d/next/Library/2020-07-26-18-17-30.bpo-41402.YRkVkp.rst D Misc/NEWS.d/next/Library/2020-07-30-14-37-15.bpo-20684.qV35GU.rst D Misc/NEWS.d/next/Library/2020-09-10-07-23-24.bpo-41730.DyKFi9.rst D Misc/NEWS.d/next/Library/2020-10-01-21-46-34.bpo-40956._tvsZ7.rst D Misc/NEWS.d/next/Library/2020-10-11-20-23-48.bpo-37449.f-t3V6.rst D Misc/NEWS.d/next/Library/2020-10-18-09-42-53.bpo-40497.CRz2sG.rst D Misc/NEWS.d/next/Library/2020-12-08-01-08-58.bpo-41818.zO8vV7.rst D Misc/NEWS.d/next/Library/2021-01-13-00-02-44.bpo-42862.Z6ACLN.rst D Misc/NEWS.d/next/Library/2021-01-16-18-36-00.bpo-33809.BiMK6V.rst D Misc/NEWS.d/next/Library/2021-01-25-21-24-55.bpo-43024.vAUrIi.rst D Misc/NEWS.d/next/Library/2021-01-31-18-24-54.bpo-43086.2_P-SH.rst D Misc/NEWS.d/next/Library/2021-02-02-20-11-14.bpo-42971.OpVoFu.rst D Misc/NEWS.d/next/Library/2021-02-04-23-16-03.bpo-30077.v6TqAi.rst D Misc/NEWS.d/next/Library/2021-02-15-21-17-46.bpo-43232.awc4yZ.rst D Misc/NEWS.d/next/Library/2021-02-15-22-14-31.bpo-43234.F-vKAT.rst D Misc/NEWS.d/next/Library/2021-02-25-08-32-06.bpo-43318.bZJw6V.rst D Misc/NEWS.d/next/Library/2021-03-03-13-32-37.bpo-43392.QQumou.rst D Misc/NEWS.d/next/Library/2021-03-24-09-40-02.bpo-43612.vMGZ4y.rst D Misc/NEWS.d/next/Library/2021-03-29-00-23-30.bpo-43650.v01tic.rst D Misc/NEWS.d/next/Library/2021-03-30-08-39-08.bpo-43666.m72tlH.rst D Misc/NEWS.d/next/Library/2021-04-15-12-02-17.bpo-43853.XXCVAp.rst D Misc/NEWS.d/next/Library/2021-04-29-00-48-00.bpo-28528.JLAVWj.rst D Misc/NEWS.d/next/Library/2021-04-30-16-58-24.bpo-43972.Y2r9lg.rst D Misc/NEWS.d/next/Library/2021-05-01-15-43-37.bpo-44002.KLT_wd.rst D Misc/NEWS.d/next/Library/2021-05-02-13-54-25.bpo-38352.N9MlhV.rst D Misc/NEWS.d/next/Library/2021-05-03-10-07-43.bpo-44018.VDyW8f.rst D Misc/NEWS.d/next/Library/2021-05-03-19-59-14.bpo-40465.1tB4Y0.rst D Misc/NEWS.d/next/Library/2021-05-05-11-44-49.bpo-36515.uOSa3q.rst D Misc/NEWS.d/next/Library/2021-05-06-16-01-55.bpo-44059.GF5r6O.rst D Misc/NEWS.d/next/Library/2021-05-07-08-39-23.bpo-44061.MvElG6.rst D Misc/NEWS.d/next/Library/2021-05-09-03-26-31.bpo-44081.A-Mrto.rst D Misc/NEWS.d/next/Library/2021-05-09-22-52-34.bpo-44089.IoANsN.rst D Misc/NEWS.d/next/Library/2021-05-10-17-45-00.bpo-44098._MoxuZ.rst D Misc/NEWS.d/next/Library/2021-05-12-16-43-21.bpo-38908.nM2_rO.rst D Misc/NEWS.d/next/Library/2021-05-13-19-07-28.bpo-37788.adeFcf.rst D Misc/NEWS.d/next/Library/2021-05-13-19-44-38.bpo-44077.04b2a4.rst D Misc/NEWS.d/next/Library/2021-05-14-16-06-02.bpo-44095.v_pLwY.rst D Misc/NEWS.d/next/Library/2021-05-16-00-00-38.bpo-44145.ko5SJ7.rst D Misc/NEWS.d/next/Library/2021-05-16-02-24-23.bpo-44142.t-XU8k.rst D Misc/NEWS.d/next/Library/2021-05-16-11-57-38.bpo-44150.xAhhik.rst D Misc/NEWS.d/next/Library/2021-05-16-17-48-24.bpo-33433.MyzO71.rst D Misc/NEWS.d/next/Library/2021-05-17-07-24-24.bpo-44154.GRI5bf.rst D Misc/NEWS.d/next/Library/2021-05-17-21-05-06.bpo-4928.Ot2yjO.rst D Misc/NEWS.d/next/Library/2021-05-18-00-17-21.bpo-27334.32EJZi.rst D Misc/NEWS.d/next/Library/2021-05-21-12-12-35.bpo-43643.GWnmcF.rst D Misc/NEWS.d/next/Library/2021-05-21-21-23-43.bpo-44210.5afQ3K.rst D Misc/NEWS.d/next/Library/2021-05-25-23-26-38.bpo-43216.xTUyyX.rst D Misc/NEWS.d/next/Library/2021-05-26-13-15-51.bpo-44241.TBqej8.rst D Misc/NEWS.d/next/Library/2021-05-26-13-34-37.bpo-33693.3okzdo.rst D Misc/NEWS.d/next/Library/2021-05-26-14-50-06.bpo-38693.NkMacJ.rst D Misc/NEWS.d/next/Library/2021-05-26-22-04-40.bpo-44235.qFBYpp.rst D Misc/NEWS.d/next/Library/2021-05-28-09-43-33.bpo-44258.nh5F7R.rst D Misc/NEWS.d/next/Library/2021-05-29-01-05-43.bpo-44254.f06xDm.rst D Misc/NEWS.d/next/Library/2021-05-30-13-32-09.bpo-44260.ROEbVd.rst D Misc/NEWS.d/next/Library/2021-05-31-04-51-02.bpo-43858.r7LOu6.rst D Misc/NEWS.d/next/Library/2021-05-31-11-28-03.bpo-44246.nhmt-v.rst D Misc/NEWS.d/next/Library/2021-05-31-11-34-56.bpo-44246.yHAkF0.rst D Misc/NEWS.d/next/Library/2021-06-07-10-26-14.bpo-44242.MKeMCQ.rst D Misc/NEWS.d/next/Library/2021-06-08-17-47-38.bpo-44339.9JwMSc.rst D Misc/NEWS.d/next/Library/2021-06-09-08-32-39.bpo-44357.70Futb.rst D Misc/NEWS.d/next/Library/2021-06-09-10-08-32.bpo-35800.3hmkWw.rst D Misc/NEWS.d/next/Library/2021-06-10-07-26-12.bpo-44351.rvyf2v.rst D Misc/NEWS.d/next/Library/2021-06-10-08-35-38.bpo-44356.6oDFhO.rst D Misc/NEWS.d/next/Library/2021-06-10-15-06-47.bpo-44342.qqkGlj.rst D Misc/NEWS.d/next/Library/2021-06-10-20-07-32.bpo-44362.oVOMfd.rst D Misc/NEWS.d/next/Library/2021-06-10-21-53-46.bpo-34266.k3fxnm.rst D Misc/NEWS.d/next/Library/2021-06-12-10-08-14.bpo-44395.PcW6Sx.rst D Misc/NEWS.d/next/Library/2021-06-12-21-25-35.bpo-27827.TMWh1i.rst D Misc/NEWS.d/next/Library/2021-06-12-22-58-20.bpo-44389.WTRnoC.rst D Misc/NEWS.d/next/Library/2021-06-13-00-16-56.bpo-37880.5bTrkw.rst D Misc/NEWS.d/next/Library/2021-06-14-14-19-11.bpo-38291.ee4cSX.rst D Misc/NEWS.d/next/Library/2021-06-14-23-28-17.bpo-44422.BlWOgv.rst D Misc/NEWS.d/next/Library/2021-06-15-13-51-25.bpo-42972.UnyYo1.rst D Misc/NEWS.d/next/Library/2021-06-16-16-52-14.bpo-44434.SQS4Pg.rst D Misc/NEWS.d/next/Library/2021-06-17-15-01-51.bpo-44439.1S7QhT.rst D Misc/NEWS.d/next/Library/2021-06-17-22-39-34.bpo-44446.qwdRic.rst D Misc/NEWS.d/next/Library/2021-06-19-21-52-27.bpo-44464.U2oa-a.rst D Misc/NEWS.d/next/Library/2021-06-20-07-14-46.bpo-44458.myqCQ0.rst D Misc/NEWS.d/next/Library/2021-06-20-14-03-18.bpo-41546.lO1jXU.rst D Misc/NEWS.d/next/Library/2021-06-20-19-01-11.bpo-44404.McfrYB.rst D Misc/NEWS.d/next/Library/2021-06-21-10-46-58.bpo-44471.2QjXv_.rst D Misc/NEWS.d/next/Library/2021-06-21-12-43-04.bpo-44466.NSm6mv.rst D Misc/NEWS.d/next/Library/2021-06-22-08-43-04.bpo-44482.U9GznK.rst D Misc/NEWS.d/next/Library/2021-06-22-16-45-48.bpo-43977.bamAGF.rst D Misc/NEWS.d/next/Library/2021-06-23-01-33-01.bpo-44491.tiOlr5.rst D Misc/NEWS.d/next/Library/2021-06-23-19-02-00.bpo-44468.-klV5-.rst D Misc/NEWS.d/next/Library/2021-06-24-19-16-20.bpo-42892.qvRNhI.rst D Misc/NEWS.d/next/Library/2021-06-26-12-27-14.bpo-44516.BVyX_y.rst D Misc/NEWS.d/next/Library/2021-06-29-07-27-08.bpo-43625.ZlAxhp.rst D Misc/NEWS.d/next/Library/2021-06-29-21-17-17.bpo-44461.acqRnV.rst D Misc/NEWS.d/next/Library/2021-06-30-11-34-35.bpo-44539.nP0Xi4.rst D Misc/NEWS.d/next/Library/2021-06-30-13-29-49.bpo-34798.t7FCa0.rst D Misc/NEWS.d/next/Library/2021-07-02-18-17-56.bpo-44554.aBUmJo.rst D Misc/NEWS.d/next/Library/2021-07-04-11-33-34.bpo-41249.sHdwBE.rst D Misc/NEWS.d/next/Library/2021-07-04-21-16-53.bpo-44558.cm7Slv.rst D Misc/NEWS.d/next/Library/2021-07-05-18-13-25.bpo-44566.o51Bd1.rst D Misc/NEWS.d/next/Library/2021-07-08-12-22-54.bpo-44569.KZ02v9.rst D Misc/NEWS.d/next/Library/2021-07-09-07-14-37.bpo-41928.Q1jMrr.rst D Misc/NEWS.d/next/Library/2021-07-10-19-55-13.bpo-42799.ad4tq8.rst D Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst D Misc/NEWS.d/next/Library/2021-07-13-09-01-33.bpo-44608.R3IcM1.rst D Misc/NEWS.d/next/Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst D Misc/NEWS.d/next/Library/2021-07-16-08-57-27.bpo-44638.EwYKne.rst D Misc/NEWS.d/next/Library/2021-07-16-13-40-31.bpo-40897.aveAre.rst D Misc/NEWS.d/next/Library/2021-07-19-14-04-42.bpo-44524.Nbf2JC.rst D Misc/NEWS.d/next/Library/2021-07-19-18-45-00.bpo-44678.YMEAu0.rst D Misc/NEWS.d/next/Library/2021-07-19-22-43-15.bpo-44353.HF81_Q.rst D Misc/NEWS.d/next/Library/2021-07-20-00-11-47.bpo-44682.3m2qVV.rst D Misc/NEWS.d/next/Library/2021-07-20-18-34-16.bpo-44353.ATuYq4.rst D Misc/NEWS.d/next/Library/2021-07-20-19-35-49.bpo-44686.ucCGhu.rst D Misc/NEWS.d/next/Library/2021-07-20-21-51-35.bpo-42854.ThuDMI.rst D Misc/NEWS.d/next/Library/2021-07-20-22-03-24.bpo-44690.tV7Zjg.rst D Misc/NEWS.d/next/Library/2021-07-20-23-28-26.bpo-44688.buFgz3.rst D Misc/NEWS.d/next/Library/2021-07-21-10-43-22.bpo-44666.CEThkv.rst D Misc/NEWS.d/next/Library/2021-07-21-23-16-30.bpo-44704.iqHLxQ.rst D Misc/NEWS.d/next/Library/2021-07-24-02-17-59.bpo-44720.shU5Qm.rst D Misc/NEWS.d/next/Library/2021-07-25-08-17-55.bpo-42378.WIhUZK.rst D Misc/NEWS.d/next/Library/2021-07-27-12-06-19.bpo-44747.epUzZz.rst D Misc/NEWS.d/next/Library/2021-07-27-22-11-29.bpo-44752._bvbrZ.rst D Misc/NEWS.d/next/Library/2021-07-28-22-53-18.bpo-44771.BvLdnU.rst D Misc/NEWS.d/next/Library/2021-07-30-23-27-30.bpo-44667.tu0Xrv.rst D Misc/NEWS.d/next/Library/2021-07-31-08-45-31.bpo-44784.fIMIDS.rst D Misc/NEWS.d/next/Library/2021-07-31-20-28-20.bpo-44793.woaQSg.rst D Misc/NEWS.d/next/Library/2021-08-01-19-49-09.bpo-27275.QsvE0k.rst D Misc/NEWS.d/next/Library/2021-08-02-14-37-32.bpo-44806.wOW_Qn.rst D Misc/NEWS.d/next/Library/2021-08-03-20-37-45.bpo-44801.i49Aug.rst D Misc/NEWS.d/next/Library/2021-08-04-12-29-00.bpo-44822.zePNXA.rst D Misc/NEWS.d/next/Library/2021-08-05-14-59-39.bpo-44839.MURNL9.rst D Misc/NEWS.d/next/Library/2021-08-05-18-20-17.bpo-44524.9T1tfe.rst D Misc/NEWS.d/next/Library/2021-08-06-09-43-50.bpo-44605.q4YSBZ.rst D Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst D Misc/NEWS.d/next/Library/2021-08-06-19-15-52.bpo-44581.oFDBTB.rst D Misc/NEWS.d/next/Library/2021-08-07-17-28-56.bpo-44859.CCopjk.rst D Misc/NEWS.d/next/Library/2021-08-07-22-51-32.bpo-44860.PTvRrU.rst D Misc/NEWS.d/next/Library/2021-08-09-13-17-10.bpo-38956.owWLNv.rst D Misc/NEWS.d/next/Library/2021-08-10-16-57-10.bpo-44524.dk9QX4.rst D Misc/NEWS.d/next/Library/2021-08-12-16-22-16.bpo-41322.utscTd.rst D Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst D Misc/NEWS.d/next/Library/2021-08-17-16-01-44.bpo-44935.roUl0G.rst D Misc/NEWS.d/next/Library/2021-08-18-10-36-14.bpo-39039.A63LYh.rst D Misc/NEWS.d/next/Library/2021-08-19-15-03-54.bpo-44955.1mxFQS.rst D Misc/NEWS.d/next/Library/2021-08-19-23-49-10.bpo-42255.ofe3ms.rst D Misc/NEWS.d/next/Library/2021-08-22-13-25-17.bpo-44019.BN8HDy.rst D Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst D Misc/NEWS.d/next/Library/2021-08-25-10-28-49.bpo-43613.WkYmI0.rst D Misc/NEWS.d/next/Library/2021-08-25-20-18-31.bpo-39218.BlO6jW.rst D Misc/NEWS.d/next/Library/2021-08-26-09-54-14.bpo-45010.Cn23bQ.rst D Misc/NEWS.d/next/Library/2021-08-26-16-25-48.bpo-45001.tn_dKp.rst D Misc/NEWS.d/next/Library/2021-08-27-19-01-23.bpo-45030.tAmBbY.rst D Misc/NEWS.d/next/Library/2021-08-27-23-40-51.bpo-43913.Uo1Gt5.rst D Misc/NEWS.d/next/Library/2021-08-28-13-00-12.bpo-45021.rReeaj.rst D Misc/NEWS.d/next/Library/2021-08-29-14-49-22.bpo-41620.WJ6PFL.rst D Misc/NEWS.d/next/Library/2021-08-30-13-55-09.bpo-31299.9QzjZs.rst D Misc/NEWS.d/next/Library/2021-09-01-15-27-00.bpo-45075.9xUpvt.rst D Misc/NEWS.d/next/Library/2021-09-02-00-18-32.bpo-40360.9nmMtB.rst D Misc/NEWS.d/next/Library/2021-09-02-00-47-14.bpo-45085.mMnaDv.rst D Misc/NEWS.d/next/Library/2021-09-02-12-42-25.bpo-45081.tOjJ1k.rst D Misc/NEWS.d/next/Library/2021-09-05-13-15-08.bpo-25894.zjbi2f.rst D Misc/NEWS.d/next/Library/2021-09-05-20-33-25.bpo-45034.62NLD5.rst D Misc/NEWS.d/next/Library/2021-09-05-21-37-28.bpo-30856.jj86y0.rst D Misc/NEWS.d/next/Library/2021-09-07-09-13-27.bpo-45124.Kw5AUs.rst D Misc/NEWS.d/next/Library/2021-09-07-14-27-39.bpo-45129.vXH0gw.rst D Misc/NEWS.d/next/Library/2021-09-07-16-33-51.bpo-45132.WI9zQY.rst D Misc/NEWS.d/next/Library/2021-09-08-01-19-31.bpo-20499.tSxx8Y.rst D Misc/NEWS.d/next/Library/2021-09-08-13-19-29.bpo-38371.y1kEfP.rst D Misc/NEWS.d/next/Library/2021-09-10-13-20-53.bpo-45162.2Jh-lq.rst D Misc/NEWS.d/next/Library/2021-09-10-21-35-53.bpo-45166.UHipXF.rst D Misc/NEWS.d/next/Library/2021-09-11-10-45-12.bpo-35474.tEY3SD.rst D Misc/NEWS.d/next/Library/2021-09-11-14-41-02.bpo-44987.Mt8DiX.rst D Misc/NEWS.d/next/Library/2021-09-11-17-46-20.bpo-45173.UptGAn.rst D Misc/NEWS.d/next/Library/2021-09-11-18-44-40.bpo-21302.QxHRpR.rst D Misc/NEWS.d/next/Library/2021-09-13-14-28-49.bpo-45168.Z1mfW4.rst D Misc/NEWS.d/next/Library/2021-09-13-14-59-01.bpo-20524.PMQ1Fh.rst D Misc/NEWS.d/next/Library/2021-09-13-19-32-58.bpo-42135.1ZAHqR.rst D Misc/NEWS.d/next/Library/2021-09-16-19-02-14.bpo-45225.xmKV4i.rst D Misc/NEWS.d/next/Library/2021-09-17-09-59-33.bpo-45228.WV1dcT.rst D Misc/NEWS.d/next/Library/2021-09-17-11-20-55.bpo-45234.qUcTVt.rst D Misc/NEWS.d/next/Library/2021-09-17-15-58-53.bpo-45183.Vv_vch.rst D Misc/NEWS.d/next/Library/2021-09-17-16-55-37.bpo-45235.sXnmPA.rst D Misc/NEWS.d/next/Library/2021-09-18-13-14-57.bpo-36674.a2k5Zb.rst D Misc/NEWS.d/next/Library/2021-09-18-16-56-33.bpo-45238.Hng_9V.rst D Misc/NEWS.d/next/Library/2021-09-20-22-46-40.bpo-21302.h56430.rst D Misc/NEWS.d/next/Library/2021-09-22-23-56-15.bpo-21302.vvQ3Su.rst D Misc/NEWS.d/next/Library/2021-09-23-22-17-26.bpo-45274.gPpa4E.rst D Misc/NEWS.d/next/Library/2021-09-24-17-20-23.bpo-1596321.3nhPUk.rst D Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst D Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst D Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst D Misc/NEWS.d/next/Security/2021-05-05-17-37-04.bpo-44022.bS3XJ9.rst D Misc/NEWS.d/next/Security/2021-05-08-11-50-46.bpo-43124.2CTM6M.rst D Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst D Misc/NEWS.d/next/Security/2021-06-29-23-40-22.bpo-41180.uTWHv_.rst D Misc/NEWS.d/next/Security/2021-07-25-20-04-54.bpo-44600.0WMldg.rst D Misc/NEWS.d/next/Security/2021-08-29-12-39-44.bpo-42278.jvmQz_.rst D Misc/NEWS.d/next/Tests/2019-09-25-18-10-10.bpo-30256.A5i76Q.rst D Misc/NEWS.d/next/Tests/2020-10-25-19-20-26.bpo-35753.2LT-hO.rst D Misc/NEWS.d/next/Tests/2021-05-04-18-10-57.bpo-42083.EMS2TK.rst D Misc/NEWS.d/next/Tests/2021-05-07-15-46-04.bpo-31904.8dk3la.rst D Misc/NEWS.d/next/Tests/2021-05-14-14-13-44.bpo-44131.YPizoC.rst D Misc/NEWS.d/next/Tests/2021-06-02-17-41-42.bpo-43921.xP7yZ4.rst D Misc/NEWS.d/next/Tests/2021-06-03-03-29-34.bpo-43921.nwH1FS.rst D Misc/NEWS.d/next/Tests/2021-06-09-15-32-05.bpo-44364.zu9Zee.rst D Misc/NEWS.d/next/Tests/2021-06-10-11-19-43.bpo-44363.-K9jD0.rst D Misc/NEWS.d/next/Tests/2021-06-18-15-19-35.bpo-44451.aj5pqE.rst D Misc/NEWS.d/next/Tests/2021-06-21-17-53-41.bpo-44287.YON57s.rst D Misc/NEWS.d/next/Tests/2021-06-26-18-37-36.bpo-44515.e9fO6f.rst D Misc/NEWS.d/next/Tests/2021-07-16-14-02-33.bpo-44647.5LzqIy.rst D Misc/NEWS.d/next/Tests/2021-07-17-11-41-20.bpo-42095.kCB7oj.rst D Misc/NEWS.d/next/Tests/2021-07-22-16-38-39.bpo-44708.SYNaac.rst D Misc/NEWS.d/next/Tests/2021-07-24-20-09-15.bpo-44734.KKsNOV.rst D Misc/NEWS.d/next/Tests/2021-08-06-00-07-15.bpo-40928.aIwb6G.rst D Misc/NEWS.d/next/Tests/2021-08-06-18-36-04.bpo-44852.sUL8YX.rst D Misc/NEWS.d/next/Tests/2021-08-13-12-11-06.bpo-44891.T9_mBT.rst D Misc/NEWS.d/next/Tests/2021-08-18-18-30-12.bpo-44949.VE5ENv.rst D Misc/NEWS.d/next/Tests/2021-08-26-14-20-44.bpo-45011.mQZdXU.rst D Misc/NEWS.d/next/Tests/2021-08-27-22-37-19.bpo-25130.ig4oJe.rst D Misc/NEWS.d/next/Tests/2021-08-30-11-54-14.bpo-45042.QMz3X8.rst D Misc/NEWS.d/next/Tests/2021-09-01-17-17-44.bpo-44895.kV7H77.rst D Misc/NEWS.d/next/Tests/2021-09-06-19-00-29.bpo-45052.yrOK3J.rst D Misc/NEWS.d/next/Tests/2021-09-08-13-01-37.bpo-44860.qXd0kx.rst D Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst D Misc/NEWS.d/next/Tests/2021-09-13-00-28-17.bpo-45156.8oomV3.rst D Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst D Misc/NEWS.d/next/Tests/2021-09-14-14-54-04.bpo-45185.qFx5I6.rst D Misc/NEWS.d/next/Tests/2021-09-15-23-32-39.bpo-45209.55ntL5.rst D Misc/NEWS.d/next/Tests/2021-09-16-17-22-35.bpo-45128.Jz6fl2.rst D Misc/NEWS.d/next/Tests/2021-09-24-10-41-49.bpo-45269.8jKEr8.rst D Misc/NEWS.d/next/Tests/2021-09-25-11-05-31.bpo-45280.3MA6lC.rst D Misc/NEWS.d/next/Tests/2021-09-30-16-54-39.bpo-40173.J_slCw.rst D Misc/NEWS.d/next/Tools-Demos/2020-02-25-18-22-09.bpo-20291.AyrDiZ.rst D Misc/NEWS.d/next/Tools-Demos/2021-05-08-13-57-00.bpo-44074.F09kCK.rst D Misc/NEWS.d/next/Tools-Demos/2021-07-01-22-21-25.bpo-43425.t65len.rst D Misc/NEWS.d/next/Tools-Demos/2021-08-22-11-45-31.bpo-44978.QupKV3.rst D Misc/NEWS.d/next/Tools-Demos/2021-08-26-11-57-31.bpo-44967.UT1RMV.rst D Misc/NEWS.d/next/Tools-Demos/2021-09-14-11-44-26.bpo-44786.DU0LC0.rst D Misc/NEWS.d/next/Windows/2020-04-13-15-20-28.bpo-40263.1KKEbu.rst D Misc/NEWS.d/next/Windows/2021-01-01-21-21-03.bpo-42686.G_f-TC.rst D Misc/NEWS.d/next/Windows/2021-06-06-16-36-13.bpo-41299.Rg-vb_.rst D Misc/NEWS.d/next/Windows/2021-07-07-21-07-18.bpo-44582.4Mm6Hh.rst D Misc/NEWS.d/next/Windows/2021-07-13-15-32-49.bpo-44572.gXvhDc.rst D Misc/NEWS.d/next/Windows/2021-08-06-10-11-07.bpo-44848.ib3Jcz.rst D Misc/NEWS.d/next/Windows/2021-08-27-23-50-02.bpo-45007.NIBlVG.rst D Misc/NEWS.d/next/Windows/2021-09-03-18-05-21.bpo-45022.bgpD_r.rst D Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst D Misc/NEWS.d/next/macOS/2021-03-29-21-11-23.bpo-34932.f3Hdyd.rst D Misc/NEWS.d/next/macOS/2021-05-24-21-15-41.bpo-43109.npKJ9c.rst D Misc/NEWS.d/next/macOS/2021-07-12-15-42-02.bpo-41972.yUjE8j.rst D Misc/NEWS.d/next/macOS/2021-07-20-22-27-01.bpo-44689.mmT_xH.rst D Misc/NEWS.d/next/macOS/2021-08-06-10-08-41.bpo-44848.0uYXsE.rst D Misc/NEWS.d/next/macOS/2021-08-27-16-55-10.bpo-34602.ZjHsYJ.rst D Misc/NEWS.d/next/macOS/2021-08-30-00-04-10.bpo-45007.pixqUB.rst M Include/patchlevel.h M Lib/pydoc_data/topics.py M README.rst diff --git a/Include/patchlevel.h b/Include/patchlevel.h index f37c4d48e3760..28a081d0afa31 100644 --- a/Include/patchlevel.h +++ b/Include/patchlevel.h @@ -20,10 +20,10 @@ #define PY_MINOR_VERSION 11 #define PY_MICRO_VERSION 0 #define PY_RELEASE_LEVEL PY_RELEASE_LEVEL_ALPHA -#define PY_RELEASE_SERIAL 0 +#define PY_RELEASE_SERIAL 1 /* Version as a string */ -#define PY_VERSION "3.11.0a0" +#define PY_VERSION "3.11.0a1" /*--end constants--*/ /* Version as a single 4-byte hex number, e.g. 0x010502B2 == 1.5.2b2. diff --git a/Lib/pydoc_data/topics.py b/Lib/pydoc_data/topics.py index 40f7a50128619..eb523370c58d7 100644 --- a/Lib/pydoc_data/topics.py +++ b/Lib/pydoc_data/topics.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Autogenerated by Sphinx on Mon Apr 5 17:39:41 2021 +# Autogenerated by Sphinx on Tue Oct 5 13:43:52 2021 topics = {'assert': 'The "assert" statement\n' '**********************\n' '\n' @@ -551,13 +551,65 @@ 'exception.\n' ' That new exception causes the old one to be lost.\n' '\n' - '[2] A string literal appearing as the first statement in the ' + '[2] In pattern matching, a sequence is defined as one of the\n' + ' following:\n' + '\n' + ' * a class that inherits from "collections.abc.Sequence"\n' + '\n' + ' * a Python class that has been registered as\n' + ' "collections.abc.Sequence"\n' + '\n' + ' * a builtin class that has its (CPython) ' + '"Py_TPFLAGS_SEQUENCE"\n' + ' bit set\n' + '\n' + ' * a class that inherits from any of the above\n' + '\n' + ' The following standard library classes are sequences:\n' + '\n' + ' * "array.array"\n' + '\n' + ' * "collections.deque"\n' + '\n' + ' * "list"\n' + '\n' + ' * "memoryview"\n' + '\n' + ' * "range"\n' + '\n' + ' * "tuple"\n' + '\n' + ' Note:\n' + '\n' + ' Subject values of type "str", "bytes", and "bytearray" do ' + 'not\n' + ' match sequence patterns.\n' + '\n' + '[3] In pattern matching, a mapping is defined as one of the ' + 'following:\n' + '\n' + ' * a class that inherits from "collections.abc.Mapping"\n' + '\n' + ' * a Python class that has been registered as\n' + ' "collections.abc.Mapping"\n' + '\n' + ' * a builtin class that has its (CPython) ' + '"Py_TPFLAGS_MAPPING"\n' + ' bit set\n' + '\n' + ' * a class that inherits from any of the above\n' + '\n' + ' The standard library classes "dict" and ' + '"types.MappingProxyType"\n' + ' are mappings.\n' + '\n' + '[4] A string literal appearing as the first statement in the ' 'function\n' ' body is transformed into the function?s "__doc__" attribute ' 'and\n' ' therefore the function?s *docstring*.\n' '\n' - '[3] A string literal appearing as the first statement in the class\n' + '[5] A string literal appearing as the first statement in the class\n' ' body is transformed into the namespace?s "__doc__" item and\n' ' therefore the class?s *docstring*.\n', 'atom-identifiers': 'Identifiers (Names)\n' @@ -884,32 +936,6 @@ '*instance* of the\n' ' owner class.\n' '\n' - 'object.__set_name__(self, owner, name)\n' - '\n' - ' Called at the time the owning class *owner* is ' - 'created. The\n' - ' descriptor has been assigned to *name*.\n' - '\n' - ' Note:\n' - '\n' - ' "__set_name__()" is only called implicitly as part ' - 'of the "type"\n' - ' constructor, so it will need to be called ' - 'explicitly with the\n' - ' appropriate parameters when a descriptor is added ' - 'to a class\n' - ' after initial creation:\n' - '\n' - ' class A:\n' - ' pass\n' - ' descr = custom_descriptor()\n' - ' A.attr = descr\n' - " descr.__set_name__(A, 'attr')\n" - '\n' - ' See Creating the class object for more details.\n' - '\n' - ' New in version 3.6.\n' - '\n' 'The attribute "__objclass__" is interpreted by the ' '"inspect" module as\n' 'specifying the class where this object was defined ' @@ -988,9 +1014,9 @@ '\n' 'For instance bindings, the precedence of descriptor ' 'invocation depends\n' - 'on the which descriptor methods are defined. A ' - 'descriptor can define\n' - 'any combination of "__get__()", "__set__()" and ' + 'on which descriptor methods are defined. A descriptor ' + 'can define any\n' + 'combination of "__get__()", "__set__()" and ' '"__delete__()". If it\n' 'does not define "__get__()", then accessing the ' 'attribute will return\n' @@ -1261,6 +1287,10 @@ 'In the latter case, sequence repetition is performed; a negative\n' 'repetition factor yields an empty sequence.\n' '\n' + 'This operation can be customized using the special "__mul__()" ' + 'and\n' + '"__rmul__()" methods.\n' + '\n' 'The "@" (at) operator is intended to be used for matrix\n' 'multiplication. No builtin Python types implement this operator.\n' '\n' @@ -1276,6 +1306,10 @@ 'result. Division by zero raises the "ZeroDivisionError" ' 'exception.\n' '\n' + 'This operation can be customized using the special "__truediv__()" ' + 'and\n' + '"__floordiv__()" methods.\n' + '\n' 'The "%" (modulo) operator yields the remainder from the division ' 'of\n' 'the first argument by the second. The numeric arguments are ' @@ -1307,6 +1341,10 @@ 'string formatting is described in the Python Library Reference,\n' 'section printf-style String Formatting.\n' '\n' + 'The *modulo* operation can be customized using the special ' + '"__mod__()"\n' + 'method.\n' + '\n' 'The floor division operator, the modulo operator, and the ' '"divmod()"\n' 'function are not defined for complex numbers. Instead, convert to ' @@ -1321,9 +1359,16 @@ 'and then added together. In the latter case, the sequences are\n' 'concatenated.\n' '\n' + 'This operation can be customized using the special "__add__()" ' + 'and\n' + '"__radd__()" methods.\n' + '\n' 'The "-" (subtraction) operator yields the difference of its ' 'arguments.\n' - 'The numeric arguments are first converted to a common type.\n', + 'The numeric arguments are first converted to a common type.\n' + '\n' + 'This operation can be customized using the special "__sub__()" ' + 'method.\n', 'bitwise': 'Binary bitwise operations\n' '*************************\n' '\n' @@ -1336,14 +1381,18 @@ '\n' 'The "&" operator yields the bitwise AND of its arguments, which ' 'must\n' - 'be integers.\n' + 'be integers or one of them must be a custom object overriding\n' + '"__and__()" or "__rand__()" special methods.\n' '\n' 'The "^" operator yields the bitwise XOR (exclusive OR) of its\n' - 'arguments, which must be integers.\n' + 'arguments, which must be integers or one of them must be a ' + 'custom\n' + 'object overriding "__xor__()" or "__rxor__()" special methods.\n' '\n' 'The "|" operator yields the bitwise (inclusive) OR of its ' 'arguments,\n' - 'which must be integers.\n', + 'which must be integers or one of them must be a custom object\n' + 'overriding "__or__()" or "__ror__()" special methods.\n', 'bltin-code-objects': 'Code Objects\n' '************\n' '\n' @@ -1360,6 +1409,10 @@ 'through their "__code__" attribute. See also the ' '"code" module.\n' '\n' + 'Accessing "__code__" raises an auditing event ' + '"object.__getattr__"\n' + 'with arguments "obj" and ""__code__"".\n' + '\n' 'A code object can be executed or evaluated by passing ' 'it (instead of a\n' 'source string) to the "exec()" or "eval()" built-in ' @@ -1704,7 +1757,7 @@ 'original global namespace. (Usually, the suite contains mostly\n' 'function definitions.) When the class?s suite finishes execution, ' 'its\n' - 'execution frame is discarded but its local namespace is saved. [3] ' + 'execution frame is discarded but its local namespace is saved. [5] ' 'A\n' 'class object is then created using the inheritance list for the ' 'base\n' @@ -1785,7 +1838,11 @@ ' comp_operator ::= "<" | ">" | "==" | ">=" | "<=" | "!="\n' ' | "is" ["not"] | ["not"] "in"\n' '\n' - 'Comparisons yield boolean values: "True" or "False".\n' + 'Comparisons yield boolean values: "True" or "False". Custom ' + '*rich\n' + 'comparison methods* may return non-boolean values. In this ' + 'case Python\n' + 'will call "bool()" on such value in boolean contexts.\n' '\n' 'Comparisons can be chained arbitrarily, e.g., "x < y <= z" ' 'is\n' @@ -2381,11 +2438,11 @@ 'resulting\n' 'object is ?compatible? with the exception. An object is ' 'compatible\n' - 'with an exception if it is the class or a base class of the ' - 'exception\n' - 'object, or a tuple containing an item that is the class or a ' - 'base\n' - 'class of the exception object.\n' + 'with an exception if the object is the class or a base class of ' + 'the\n' + 'exception object, or a tuple containing an item that is the ' + 'class or a\n' + 'base class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' @@ -2694,7 +2751,7 @@ ' subject_expr ::= star_named_expression "," ' 'star_named_expressions?\n' ' | named_expression\n' - " case_block ::= 'case' patterns [guard] ':' block\n" + ' case_block ::= \'case\' patterns [guard] ":" block\n' '\n' 'Note:\n' '\n' @@ -2764,7 +2821,7 @@ 'have\n' ' happened.\n' '\n' - ' * If the guard evaluates as truthy or missing, the "block" ' + ' * If the guard evaluates as true or is missing, the "block" ' 'inside\n' ' "case_block" is executed.\n' '\n' @@ -2825,12 +2882,12 @@ '\n' '2. If the pattern succeeded, evaluate the "guard".\n' '\n' - ' * If the "guard" condition evaluates to ?truthy?, the case ' - 'block is\n' + ' * If the "guard" condition evaluates as true, the case block ' + 'is\n' ' selected.\n' '\n' - ' * If the "guard" condition evaluates to ?falsy?, the case ' - 'block is\n' + ' * If the "guard" condition evaluates as false, the case block ' + 'is\n' ' not selected.\n' '\n' ' * If the "guard" raises an exception during evaluation, the\n' @@ -3001,7 +3058,7 @@ '\n' 'A single underscore "_" is not a capture pattern (this is what ' '"!\'_\'"\n' - 'expresses). And is instead treated as a "wildcard_pattern".\n' + 'expresses). It is instead treated as a "wildcard_pattern".\n' '\n' 'In a given pattern, a given name can only be bound once. E.g. ' '"case\n' @@ -3029,7 +3086,10 @@ '\n' " wildcard_pattern ::= '_'\n" '\n' - '"_" is a soft keyword.\n' + '"_" is a soft keyword within any pattern, but only within ' + 'patterns.\n' + 'It is an identifier, as usual, even within "match" subject\n' + 'expressions, "guard"s, and "case" blocks.\n' '\n' 'In simple terms, "_" will always succeed.\n' '\n' @@ -3073,7 +3133,7 @@ 'additional\n' 'syntax. Syntax:\n' '\n' - " group_pattern ::= '(' pattern ')'\n" + ' group_pattern ::= "(" pattern ")"\n' '\n' 'In simple terms "(P)" has the same effect as "P".\n' '\n' @@ -3120,8 +3180,9 @@ 'pattern\n' 'against a subject value:\n' '\n' - '1. If the subject value is not an instance of a\n' - ' "collections.abc.Sequence" the sequence pattern fails.\n' + '1. If the subject value is not a sequence [2], the sequence ' + 'pattern\n' + ' fails.\n' '\n' '2. If the subject value is an instance of "str", "bytes" or\n' ' "bytearray" the sequence pattern fails.\n' @@ -3176,7 +3237,7 @@ 'the\n' 'following happens:\n' '\n' - '* "isinstance(, collections.abc.Sequence)"\n' + '* check "" is a sequence\n' '\n' '* "len(subject) == "\n' '\n' @@ -3210,18 +3271,19 @@ 'double star pattern must be the last subpattern in the mapping\n' 'pattern.\n' '\n' - 'Duplicate key values in mapping patterns are disallowed. (If all ' - 'key\n' - 'patterns are literal patterns this is considered a syntax ' - 'error;\n' - 'otherwise this is a runtime error and will raise "ValueError".)\n' + 'Duplicate keys in mapping patterns are disallowed. Duplicate ' + 'literal\n' + 'keys will raise a "SyntaxError". Two keys that otherwise have ' + 'the same\n' + 'value will raise a "ValueError" at runtime.\n' '\n' 'The following is the logical flow for matching a mapping ' 'pattern\n' 'against a subject value:\n' '\n' - '1. If the subject value is not an instance of\n' - ' "collections.abc.Mapping", the mapping pattern fails.\n' + '1. If the subject value is not a mapping [3],the mapping ' + 'pattern\n' + ' fails.\n' '\n' '2. If every key given in the mapping pattern is present in the ' 'subject\n' @@ -3231,7 +3293,10 @@ '\n' '3. If duplicate keys are detected in the mapping pattern, the ' 'pattern\n' - ' is considered invalid and "ValueError" is raised.\n' + ' is considered invalid. A "SyntaxError" is raised for ' + 'duplicate\n' + ' literal values; or a "ValueError" for named keys of the same ' + 'value.\n' '\n' 'Note:\n' '\n' @@ -3247,7 +3312,7 @@ 'the\n' 'following happens:\n' '\n' - '* "isinstance(, collections.abc.Mapping)"\n' + '* check "" is a mapping\n' '\n' '* "KEY1 in "\n' '\n' @@ -3293,7 +3358,9 @@ ' For a number of built-in types (specified below), a single\n' ' positional subpattern is accepted which will match the ' 'entire\n' - ' subject; for these types no keyword patterns are accepted.\n' + ' subject; for these types keyword patterns also work as for ' + 'other\n' + ' types.\n' '\n' ' If only keyword patterns are present, they are processed as\n' ' follows, one by one:\n' @@ -3324,15 +3391,14 @@ 'class\n' ' "name_or_attr" before matching:\n' '\n' - ' I. The equivalent of "getattr(cls, "__match_args__", ())" ' - 'is\n' + ' I. The equivalent of "getattr(cls, "__match_args__", ())" is\n' ' called.\n' '\n' ' * If this raises an exception, the exception bubbles up.\n' '\n' - ' * If the returned value is not a list or tuple, the ' - 'conversion\n' - ' fails and "TypeError" is raised.\n' + ' * If the returned value is not a tuple, the conversion ' + 'fails and\n' + ' "TypeError" is raised.\n' '\n' ' * If there are more positional patterns than\n' ' "len(cls.__match_args__)", "TypeError" is raised.\n' @@ -3426,7 +3492,6 @@ ' decorators ::= decorator+\n' ' decorator ::= "@" assignment_expression ' 'NEWLINE\n' - ' dotted_name ::= identifier ("." identifier)*\n' ' parameter_list ::= defparameter ("," ' 'defparameter)* "," "/" ["," [parameter_list_no_posonly]]\n' ' | parameter_list_no_posonly\n' @@ -3451,7 +3516,7 @@ '\n' 'The function definition does not execute the function body; this ' 'gets\n' - 'executed only when the function is called. [2]\n' + 'executed only when the function is called. [4]\n' '\n' 'A function definition may be wrapped by one or more *decorator*\n' 'expressions. Decorator expressions are evaluated when the ' @@ -3526,7 +3591,7 @@ 'Calls.\n' 'A function call always assigns values to all parameters ' 'mentioned in\n' - 'the parameter list, either from position arguments, from ' + 'the parameter list, either from positional arguments, from ' 'keyword\n' 'arguments, or from default values. If the form ?"*identifier"? ' 'is\n' @@ -3538,8 +3603,14 @@ 'new\n' 'empty mapping of the same type. Parameters after ?"*"? or\n' '?"*identifier"? are keyword-only parameters and may only be ' - 'passed\n' - 'used keyword arguments.\n' + 'passed by\n' + 'keyword arguments. Parameters before ?"/"? are positional-only\n' + 'parameters and may only be passed by positional arguments.\n' + '\n' + 'Changed in version 3.8: The "/" function parameter syntax may be ' + 'used\n' + 'to indicate positional-only parameters. See **PEP 570** for ' + 'details.\n' '\n' 'Parameters may have an *annotation* of the form ?": ' 'expression"?\n' @@ -3552,11 +3623,20 @@ 'parameter list. These annotations can be any valid Python ' 'expression.\n' 'The presence of annotations does not change the semantics of a\n' - 'function. The annotation values are available as string values ' - 'in a\n' + 'function. The annotation values are available as values of a\n' 'dictionary keyed by the parameters? names in the ' '"__annotations__"\n' - 'attribute of the function object.\n' + 'attribute of the function object. If the "annotations" import ' + 'from\n' + '"__future__" is used, annotations are preserved as strings at ' + 'runtime\n' + 'which enables postponed evaluation. Otherwise, they are ' + 'evaluated\n' + 'when the function definition is executed. In this case ' + 'annotations\n' + 'may be evaluated in a different order than they appear in the ' + 'source\n' + 'code.\n' '\n' 'It is also possible to create anonymous functions (functions not ' 'bound\n' @@ -3641,7 +3721,7 @@ 'function definitions.) When the class?s suite finishes ' 'execution, its\n' 'execution frame is discarded but its local namespace is saved. ' - '[3] A\n' + '[5] A\n' 'class object is then created using the inheritance list for the ' 'base\n' 'classes and the saved local namespace for the attribute ' @@ -3845,13 +3925,65 @@ 'exception.\n' ' That new exception causes the old one to be lost.\n' '\n' - '[2] A string literal appearing as the first statement in the ' + '[2] In pattern matching, a sequence is defined as one of the\n' + ' following:\n' + '\n' + ' * a class that inherits from "collections.abc.Sequence"\n' + '\n' + ' * a Python class that has been registered as\n' + ' "collections.abc.Sequence"\n' + '\n' + ' * a builtin class that has its (CPython) ' + '"Py_TPFLAGS_SEQUENCE"\n' + ' bit set\n' + '\n' + ' * a class that inherits from any of the above\n' + '\n' + ' The following standard library classes are sequences:\n' + '\n' + ' * "array.array"\n' + '\n' + ' * "collections.deque"\n' + '\n' + ' * "list"\n' + '\n' + ' * "memoryview"\n' + '\n' + ' * "range"\n' + '\n' + ' * "tuple"\n' + '\n' + ' Note:\n' + '\n' + ' Subject values of type "str", "bytes", and "bytearray" do ' + 'not\n' + ' match sequence patterns.\n' + '\n' + '[3] In pattern matching, a mapping is defined as one of the ' + 'following:\n' + '\n' + ' * a class that inherits from "collections.abc.Mapping"\n' + '\n' + ' * a Python class that has been registered as\n' + ' "collections.abc.Mapping"\n' + '\n' + ' * a builtin class that has its (CPython) ' + '"Py_TPFLAGS_MAPPING"\n' + ' bit set\n' + '\n' + ' * a class that inherits from any of the above\n' + '\n' + ' The standard library classes "dict" and ' + '"types.MappingProxyType"\n' + ' are mappings.\n' + '\n' + '[4] A string literal appearing as the first statement in the ' 'function\n' ' body is transformed into the function?s "__doc__" attribute ' 'and\n' ' therefore the function?s *docstring*.\n' '\n' - '[3] A string literal appearing as the first statement in the ' + '[5] A string literal appearing as the first statement in the ' 'class\n' ' body is transformed into the namespace?s "__doc__" item and\n' ' therefore the class?s *docstring*.\n', @@ -3989,13 +4121,13 @@ '\n' ' If "__new__()" is invoked during object construction and ' 'it returns\n' - ' an instance or subclass of *cls*, then the new ' - 'instance?s\n' - ' "__init__()" method will be invoked like ' - '"__init__(self[, ...])",\n' - ' where *self* is the new instance and the remaining ' - 'arguments are\n' - ' the same as were passed to the object constructor.\n' + ' an instance of *cls*, then the new instance?s ' + '"__init__()" method\n' + ' will be invoked like "__init__(self[, ...])", where ' + '*self* is the\n' + ' new instance and the remaining arguments are the same as ' + 'were\n' + ' passed to the object constructor.\n' '\n' ' If "__new__()" does not return an instance of *cls*, ' 'then the new\n' @@ -4703,13 +4835,18 @@ '\n' 'If a file ".pdbrc" exists in the user?s home directory or in ' 'the\n' - 'current directory, it is read in and executed as if it had been ' - 'typed\n' - 'at the debugger prompt. This is particularly useful for ' - 'aliases. If\n' - 'both files exist, the one in the home directory is read first ' - 'and\n' - 'aliases defined there can be overridden by the local file.\n' + 'current directory, it is read with "\'utf-8\'" encoding and ' + 'executed as\n' + 'if it had been typed at the debugger prompt. This is ' + 'particularly\n' + 'useful for aliases. If both files exist, the one in the home\n' + 'directory is read first and aliases defined there can be ' + 'overridden by\n' + 'the local file.\n' + '\n' + 'Changed in version 3.11: ".pdbrc" is now read with "\'utf-8\'" ' + 'encoding.\n' + 'Previously, it was read with the system locale encoding.\n' '\n' 'Changed in version 3.2: ".pdbrc" can now contain commands that\n' 'continue debugging, such as "continue" or "next". Previously, ' @@ -6075,19 +6212,19 @@ 'complex\n' 'types. For integers, when binary, octal, or hexadecimal ' 'output is\n' - 'used, this option adds the prefix respective "\'0b\'", ' - '"\'0o\'", or "\'0x\'"\n' - 'to the output value. For float and complex the alternate ' - 'form causes\n' - 'the result of the conversion to always contain a ' - 'decimal-point\n' - 'character, even if no digits follow it. Normally, a ' - 'decimal-point\n' - 'character appears in the result of these conversions only ' - 'if a digit\n' - 'follows it. In addition, for "\'g\'" and "\'G\'" ' - 'conversions, trailing\n' - 'zeros are not removed from the result.\n' + 'used, this option adds the respective prefix "\'0b\'", ' + '"\'0o\'", "\'0x\'",\n' + 'or "\'0X\'" to the output value. For float and complex the ' + 'alternate\n' + 'form causes the result of the conversion to always contain ' + 'a decimal-\n' + 'point character, even if no digits follow it. Normally, a ' + 'decimal-\n' + 'point character appears in the result of these conversions ' + 'only if a\n' + 'digit follows it. In addition, for "\'g\'" and "\'G\'" ' + 'conversions,\n' + 'trailing zeros are not removed from the result.\n' '\n' 'The "\',\'" option signals the use of a comma for a ' 'thousands separator.\n' @@ -6204,8 +6341,12 @@ '+-----------+------------------------------------------------------------+\n' ' | "\'X\'" | Hex format. Outputs the number in base ' '16, using upper- |\n' - ' | | case letters for the digits above ' - '9. |\n' + ' | | case letters for the digits above 9. In ' + 'case "\'#\'" is |\n' + ' | | specified, the prefix "\'0x\'" will be ' + 'upper-cased to "\'0X\'" |\n' + ' | | as ' + 'well. |\n' ' ' '+-----------+------------------------------------------------------------+\n' ' | "\'n\'" | Number. This is the same as "\'d\'", ' @@ -6562,7 +6703,6 @@ ' decorators ::= decorator+\n' ' decorator ::= "@" assignment_expression ' 'NEWLINE\n' - ' dotted_name ::= identifier ("." identifier)*\n' ' parameter_list ::= defparameter ("," ' 'defparameter)* "," "/" ["," [parameter_list_no_posonly]]\n' ' | parameter_list_no_posonly\n' @@ -6587,7 +6727,7 @@ '\n' 'The function definition does not execute the function body; this ' 'gets\n' - 'executed only when the function is called. [2]\n' + 'executed only when the function is called. [4]\n' '\n' 'A function definition may be wrapped by one or more *decorator*\n' 'expressions. Decorator expressions are evaluated when the ' @@ -6662,7 +6802,7 @@ 'Calls.\n' 'A function call always assigns values to all parameters ' 'mentioned in\n' - 'the parameter list, either from position arguments, from ' + 'the parameter list, either from positional arguments, from ' 'keyword\n' 'arguments, or from default values. If the form ?"*identifier"? ' 'is\n' @@ -6674,8 +6814,14 @@ 'new\n' 'empty mapping of the same type. Parameters after ?"*"? or\n' '?"*identifier"? are keyword-only parameters and may only be ' - 'passed\n' - 'used keyword arguments.\n' + 'passed by\n' + 'keyword arguments. Parameters before ?"/"? are positional-only\n' + 'parameters and may only be passed by positional arguments.\n' + '\n' + 'Changed in version 3.8: The "/" function parameter syntax may be ' + 'used\n' + 'to indicate positional-only parameters. See **PEP 570** for ' + 'details.\n' '\n' 'Parameters may have an *annotation* of the form ?": ' 'expression"?\n' @@ -6688,11 +6834,20 @@ 'parameter list. These annotations can be any valid Python ' 'expression.\n' 'The presence of annotations does not change the semantics of a\n' - 'function. The annotation values are available as string values ' - 'in a\n' + 'function. The annotation values are available as values of a\n' 'dictionary keyed by the parameters? names in the ' '"__annotations__"\n' - 'attribute of the function object.\n' + 'attribute of the function object. If the "annotations" import ' + 'from\n' + '"__future__" is used, annotations are preserved as strings at ' + 'runtime\n' + 'which enables postponed evaluation. Otherwise, they are ' + 'evaluated\n' + 'when the function definition is executed. In this case ' + 'annotations\n' + 'may be evaluated in a different order than they appear in the ' + 'source\n' + 'code.\n' '\n' 'It is also possible to create anonymous functions (functions not ' 'bound\n' @@ -6909,8 +7064,8 @@ '\n' 'A non-normative HTML file listing all valid identifier ' 'characters for\n' - 'Unicode 4.1 can be found at\n' - 'https://www.unicode.org/Public/13.0.0/ucd/DerivedCoreProperties.txt\n' + 'Unicode 14.0.0 can be found at\n' + 'https://www.unicode.org/Public/14.0.0/ucd/DerivedCoreProperties.txt\n' '\n' '\n' 'Keywords\n' @@ -7051,7 +7206,7 @@ ' | "from" relative_module "import" "(" ' 'identifier ["as" identifier]\n' ' ("," identifier ["as" identifier])* [","] ")"\n' - ' | "from" module "import" "*"\n' + ' | "from" relative_module "import" "*"\n' ' module ::= (identifier ".")* identifier\n' ' relative_module ::= "."* module | "."+\n' '\n' @@ -7395,10 +7550,7 @@ 'lambda': 'Lambdas\n' '*******\n' '\n' - ' lambda_expr ::= "lambda" [parameter_list] ":" ' - 'expression\n' - ' lambda_expr_nocond ::= "lambda" [parameter_list] ":" ' - 'expression_nocond\n' + ' lambda_expr ::= "lambda" [parameter_list] ":" expression\n' '\n' 'Lambda expressions (sometimes called lambda forms) are used to ' 'create\n' @@ -7715,11 +7867,11 @@ 'instance, to\n' ' evaluate the expression "x + y", where *x* is an ' 'instance of a\n' - ' class that has an "__add__()" method, "x.__add__(y)" is ' - 'called.\n' - ' The "__divmod__()" method should be the equivalent to ' - 'using\n' - ' "__floordiv__()" and "__mod__()"; it should not be ' + ' class that has an "__add__()" method, ' + '"type(x).__add__(x, y)" is\n' + ' called. The "__divmod__()" method should be the ' + 'equivalent to\n' + ' using "__floordiv__()" and "__mod__()"; it should not be ' 'related to\n' ' "__truediv__()". Note that "__pow__()" should be ' 'defined to accept\n' @@ -7760,9 +7912,9 @@ 'expression "x -\n' ' y", where *y* is an instance of a class that has an ' '"__rsub__()"\n' - ' method, "y.__rsub__(x)" is called if "x.__sub__(y)" ' - 'returns\n' - ' *NotImplemented*.\n' + ' method, "type(y).__rsub__(y, x)" is called if ' + '"type(x).__sub__(x,\n' + ' y)" returns *NotImplemented*.\n' '\n' ' Note that ternary "pow()" will not try calling ' '"__rpow__()" (the\n' @@ -8009,8 +8161,8 @@ '\n' 'The following table summarizes the operator precedence ' 'in Python, from\n' - 'lowest precedence (least binding) to highest precedence ' - '(most\n' + 'highest precedence (most binding) to lowest precedence ' + '(least\n' 'binding). Operators in the same box have the same ' 'precedence. Unless\n' 'the syntax is explicitly given, operators are binary. ' @@ -8029,71 +8181,71 @@ '| Operator | ' 'Description |\n' '|=================================================|=======================================|\n' - '| ":=" | ' - 'Assignment expression |\n' - '+-------------------------------------------------+---------------------------------------+\n' - '| "lambda" | ' - 'Lambda expression |\n' + '| "(expressions...)", "[expressions...]", "{key: | ' + 'Binding or parenthesized expression, |\n' + '| value...}", "{expressions...}" | list ' + 'display, dictionary display, set |\n' + '| | ' + 'display |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "if" ? "else" | ' - 'Conditional expression |\n' + '| "x[index]", "x[index:index]", | ' + 'Subscription, slicing, call, |\n' + '| "x(arguments...)", "x.attribute" | ' + 'attribute reference |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "or" | ' - 'Boolean OR |\n' + '| "await" "x" | ' + 'Await expression |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "and" | ' - 'Boolean AND |\n' + '| "**" | ' + 'Exponentiation [5] |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "not" "x" | ' - 'Boolean NOT |\n' + '| "+x", "-x", "~x" | ' + 'Positive, negative, bitwise NOT |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "in", "not in", "is", "is not", "<", "<=", ">", | ' - 'Comparisons, including membership |\n' - '| ">=", "!=", "==" | ' - 'tests and identity tests |\n' + '| "*", "@", "/", "//", "%" | ' + 'Multiplication, matrix |\n' + '| | ' + 'multiplication, division, floor |\n' + '| | ' + 'division, remainder [6] |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "|" | ' - 'Bitwise OR |\n' + '| "+", "-" | ' + 'Addition and subtraction |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "^" | ' - 'Bitwise XOR |\n' + '| "<<", ">>" | ' + 'Shifts |\n' '+-------------------------------------------------+---------------------------------------+\n' '| "&" | ' 'Bitwise AND |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "<<", ">>" | ' - 'Shifts |\n' + '| "^" | ' + 'Bitwise XOR |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "+", "-" | ' - 'Addition and subtraction |\n' + '| "|" | ' + 'Bitwise OR |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "*", "@", "/", "//", "%" | ' - 'Multiplication, matrix |\n' - '| | ' - 'multiplication, division, floor |\n' - '| | ' - 'division, remainder [5] |\n' + '| "in", "not in", "is", "is not", "<", "<=", ">", | ' + 'Comparisons, including membership |\n' + '| ">=", "!=", "==" | ' + 'tests and identity tests |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "+x", "-x", "~x" | ' - 'Positive, negative, bitwise NOT |\n' + '| "not" "x" | ' + 'Boolean NOT |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "**" | ' - 'Exponentiation [6] |\n' + '| "and" | ' + 'Boolean AND |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "await" "x" | ' - 'Await expression |\n' + '| "or" | ' + 'Boolean OR |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "x[index]", "x[index:index]", | ' - 'Subscription, slicing, call, |\n' - '| "x(arguments...)", "x.attribute" | ' - 'attribute reference |\n' + '| "if" ? "else" | ' + 'Conditional expression |\n' '+-------------------------------------------------+---------------------------------------+\n' - '| "(expressions...)", "[expressions...]", "{key: | ' - 'Binding or parenthesized expression, |\n' - '| value...}", "{expressions...}" | list ' - 'display, dictionary display, set |\n' - '| | ' - 'display |\n' + '| "lambda" | ' + 'Lambda expression |\n' + '+-------------------------------------------------+---------------------------------------+\n' + '| ":=" | ' + 'Assignment expression |\n' '+-------------------------------------------------+---------------------------------------+\n' '\n' '-[ Footnotes ]-\n' @@ -8174,14 +8326,14 @@ 'Check their\n' ' documentation for more info.\n' '\n' - '[5] The "%" operator is also used for string formatting; ' - 'the same\n' - ' precedence applies.\n' - '\n' - '[6] The power operator "**" binds less tightly than an ' + '[5] The power operator "**" binds less tightly than an ' 'arithmetic or\n' ' bitwise unary operator on its right, that is, ' - '"2**-1" is "0.5".\n', + '"2**-1" is "0.5".\n' + '\n' + '[6] The "%" operator is also used for string formatting; ' + 'the same\n' + ' precedence applies.\n', 'pass': 'The "pass" statement\n' '********************\n' '\n' @@ -8229,7 +8381,10 @@ '"ZeroDivisionError".\n' 'Raising a negative number to a fractional power results in a ' '"complex"\n' - 'number. (In earlier versions it raised a "ValueError".)\n', + 'number. (In earlier versions it raised a "ValueError".)\n' + '\n' + 'This operation can be customized using the special "__pow__()" ' + 'method.\n', 'raise': 'The "raise" statement\n' '*********************\n' '\n' @@ -8266,12 +8421,18 @@ '\n' 'The "from" clause is used for exception chaining: if given, the ' 'second\n' - '*expression* must be another exception class or instance, which ' - 'will\n' - 'then be attached to the raised exception as the "__cause__" ' - 'attribute\n' - '(which is writable). If the raised exception is not handled, both\n' - 'exceptions will be printed:\n' + '*expression* must be another exception class or instance. If the\n' + 'second expression is an exception instance, it will be attached to ' + 'the\n' + 'raised exception as the "__cause__" attribute (which is writable). ' + 'If\n' + 'the expression is an exception class, the class will be ' + 'instantiated\n' + 'and the resulting exception instance will be attached to the ' + 'raised\n' + 'exception as the "__cause__" attribute. If the raised exception is ' + 'not\n' + 'handled, both exceptions will be printed:\n' '\n' ' >>> try:\n' ' ... print(1 / 0)\n' @@ -8623,6 +8784,10 @@ 'the\n' 'second argument.\n' '\n' + 'This operation can be customized using the special ' + '"__lshift__()" and\n' + '"__rshift__()" methods.\n' + '\n' 'A right shift by *n* bits is defined as floor division by ' '"pow(2,n)".\n' 'A left shift by *n* bits is defined as multiplication with ' @@ -8837,13 +9002,13 @@ '\n' ' If "__new__()" is invoked during object construction and ' 'it returns\n' - ' an instance or subclass of *cls*, then the new ' - 'instance?s\n' - ' "__init__()" method will be invoked like "__init__(self[, ' - '...])",\n' - ' where *self* is the new instance and the remaining ' - 'arguments are\n' - ' the same as were passed to the object constructor.\n' + ' an instance of *cls*, then the new instance?s ' + '"__init__()" method\n' + ' will be invoked like "__init__(self[, ...])", where ' + '*self* is the\n' + ' new instance and the remaining arguments are the same as ' + 'were\n' + ' passed to the object constructor.\n' '\n' ' If "__new__()" does not return an instance of *cls*, then ' 'the new\n' @@ -9511,32 +9676,6 @@ 'of the\n' ' owner class.\n' '\n' - 'object.__set_name__(self, owner, name)\n' - '\n' - ' Called at the time the owning class *owner* is created. ' - 'The\n' - ' descriptor has been assigned to *name*.\n' - '\n' - ' Note:\n' - '\n' - ' "__set_name__()" is only called implicitly as part of ' - 'the "type"\n' - ' constructor, so it will need to be called explicitly ' - 'with the\n' - ' appropriate parameters when a descriptor is added to a ' - 'class\n' - ' after initial creation:\n' - '\n' - ' class A:\n' - ' pass\n' - ' descr = custom_descriptor()\n' - ' A.attr = descr\n' - " descr.__set_name__(A, 'attr')\n" - '\n' - ' See Creating the class object for more details.\n' - '\n' - ' New in version 3.6.\n' - '\n' 'The attribute "__objclass__" is interpreted by the "inspect" ' 'module as\n' 'specifying the class where this object was defined (setting ' @@ -9613,10 +9752,10 @@ '\n' 'For instance bindings, the precedence of descriptor ' 'invocation depends\n' - 'on the which descriptor methods are defined. A descriptor ' - 'can define\n' - 'any combination of "__get__()", "__set__()" and ' - '"__delete__()". If it\n' + 'on which descriptor methods are defined. A descriptor can ' + 'define any\n' + 'combination of "__get__()", "__set__()" and "__delete__()". ' + 'If it\n' 'does not define "__get__()", then accessing the attribute ' 'will return\n' 'the descriptor object itself unless there is a value in the ' @@ -9826,6 +9965,38 @@ '\n' ' New in version 3.6.\n' '\n' + 'When a class is created, "type.__new__()" scans the class ' + 'variables\n' + 'and makes callbacks to those with a "__set_name__()" hook.\n' + '\n' + 'object.__set_name__(self, owner, name)\n' + '\n' + ' Automatically called at the time the owning class *owner* ' + 'is\n' + ' created. The object has been assigned to *name* in that ' + 'class:\n' + '\n' + ' class A:\n' + ' x = C() # Automatically calls: x.__set_name__(A, ' + "'x')\n" + '\n' + ' If the class variable is assigned after the class is ' + 'created,\n' + ' "__set_name__()" will not be called automatically. If ' + 'needed,\n' + ' "__set_name__()" can be called directly:\n' + '\n' + ' class A:\n' + ' pass\n' + '\n' + ' c = C()\n' + ' A.x = c # The hook is not called\n' + " c.__set_name__(A, 'x') # Manually invoke the hook\n" + '\n' + ' See Creating the class object for more details.\n' + '\n' + ' New in version 3.6.\n' + '\n' '\n' 'Metaclasses\n' '-----------\n' @@ -10021,22 +10192,21 @@ 'When using the default metaclass "type", or any metaclass ' 'that\n' 'ultimately calls "type.__new__", the following additional\n' - 'customisation steps are invoked after creating the class ' + 'customization steps are invoked after creating the class ' 'object:\n' '\n' - '* first, "type.__new__" collects all of the descriptors in ' - 'the class\n' - ' namespace that define a "__set_name__()" method;\n' + '1. The "type.__new__" method collects all of the attributes ' + 'in the\n' + ' class namespace that define a "__set_name__()" method;\n' '\n' - '* second, all of these "__set_name__" methods are called ' - 'with the\n' - ' class being defined and the assigned name of that ' - 'particular\n' - ' descriptor;\n' + '2. Those "__set_name__" methods are called with the class ' + 'being\n' + ' defined and the assigned name of that particular ' + 'attribute;\n' '\n' - '* finally, the "__init_subclass__()" hook is called on the ' - 'immediate\n' - ' parent of the new class in its method resolution order.\n' + '3. The "__init_subclass__()" hook is called on the immediate ' + 'parent of\n' + ' the new class in its method resolution order.\n' '\n' 'After the class object is created, it is passed to the ' 'class\n' @@ -10437,11 +10607,11 @@ 'to\n' ' evaluate the expression "x + y", where *x* is an instance ' 'of a\n' - ' class that has an "__add__()" method, "x.__add__(y)" is ' - 'called.\n' - ' The "__divmod__()" method should be the equivalent to ' - 'using\n' - ' "__floordiv__()" and "__mod__()"; it should not be ' + ' class that has an "__add__()" method, "type(x).__add__(x, ' + 'y)" is\n' + ' called. The "__divmod__()" method should be the ' + 'equivalent to\n' + ' using "__floordiv__()" and "__mod__()"; it should not be ' 'related to\n' ' "__truediv__()". Note that "__pow__()" should be defined ' 'to accept\n' @@ -10482,9 +10652,9 @@ 'expression "x -\n' ' y", where *y* is an instance of a class that has an ' '"__rsub__()"\n' - ' method, "y.__rsub__(x)" is called if "x.__sub__(y)" ' - 'returns\n' - ' *NotImplemented*.\n' + ' method, "type(y).__rsub__(y, x)" is called if ' + '"type(x).__sub__(x,\n' + ' y)" returns *NotImplemented*.\n' '\n' ' Note that ternary "pow()" will not try calling ' '"__rpow__()" (the\n' @@ -10677,17 +10847,16 @@ '\n' 'object.__match_args__\n' '\n' - ' This class variable can be assigned a tuple or list of ' - 'strings.\n' - ' When this class is used in a class pattern with ' - 'positional\n' - ' arguments, each positional argument will be converted ' - 'into a\n' - ' keyword argument, using the corresponding value in ' - '*__match_args__*\n' - ' as the keyword. The absence of this attribute is ' - 'equivalent to\n' - ' setting it to "()".\n' + ' This class variable can be assigned a tuple of strings. ' + 'When this\n' + ' class is used in a class pattern with positional ' + 'arguments, each\n' + ' positional argument will be converted into a keyword ' + 'argument,\n' + ' using the corresponding value in *__match_args__* as the ' + 'keyword.\n' + ' The absence of this attribute is equivalent to setting it ' + 'to "()".\n' '\n' 'For example, if "MyClass.__match_args__" is "("left", ' '"center",\n' @@ -12078,10 +12247,10 @@ 'exception. For an except clause with an expression, that expression\n' 'is evaluated, and the clause matches the exception if the resulting\n' 'object is ?compatible? with the exception. An object is compatible\n' - 'with an exception if it is the class or a base class of the ' - 'exception\n' - 'object, or a tuple containing an item that is the class or a base\n' - 'class of the exception object.\n' + 'with an exception if the object is the class or a base class of the\n' + 'exception object, or a tuple containing an item that is the class or ' + 'a\n' + 'base class of the exception object.\n' '\n' 'If no except clause matches the exception, the search for an ' 'exception\n' @@ -12680,7 +12849,13 @@ '| |\n' ' | | and "\'return\'" for the ' 'return | |\n' - ' | | annotation, if provided. ' + ' | | annotation, if provided. For ' + '| |\n' + ' | | more information on working ' + '| |\n' + ' | | with this attribute, see ' + '| |\n' + ' | | Annotations Best Practices. ' '| |\n' ' ' '+---------------------------+---------------------------------+-------------+\n' @@ -12905,20 +13080,34 @@ ' Attribute assignment updates the module?s namespace dictionary,\n' ' e.g., "m.x = 1" is equivalent to "m.__dict__["x"] = 1".\n' '\n' - ' Predefined (writable) attributes: "__name__" is the module?s ' - 'name;\n' - ' "__doc__" is the module?s documentation string, or "None" if\n' - ' unavailable; "__annotations__" (optional) is a dictionary\n' - ' containing *variable annotations* collected during module body\n' - ' execution; "__file__" is the pathname of the file from which ' + ' Predefined (writable) attributes:\n' + '\n' + ' "__name__"\n' + ' The module?s name.\n' + '\n' + ' "__doc__"\n' + ' The module?s documentation string, or "None" if ' + 'unavailable.\n' + '\n' + ' "__file__"\n' + ' The pathname of the file from which the module was loaded, ' + 'if\n' + ' it was loaded from a file. The "__file__" attribute may ' + 'be\n' + ' missing for certain types of modules, such as C modules ' + 'that\n' + ' are statically linked into the interpreter. For ' + 'extension\n' + ' modules loaded dynamically from a shared library, it?s ' 'the\n' - ' module was loaded, if it was loaded from a file. The "__file__"\n' - ' attribute may be missing for certain types of modules, such as ' - 'C\n' - ' modules that are statically linked into the interpreter; for\n' - ' extension modules loaded dynamically from a shared library, it ' - 'is\n' - ' the pathname of the shared library file.\n' + ' pathname of the shared library file.\n' + '\n' + ' "__annotations__"\n' + ' A dictionary containing *variable annotations* collected\n' + ' during module body execution. For best practices on ' + 'working\n' + ' with "__annotations__", please see Annotations Best\n' + ' Practices.\n' '\n' ' Special read-only attribute: "__dict__" is the module?s ' 'namespace\n' @@ -12976,20 +13165,31 @@ 'instance\n' ' (see below).\n' '\n' - ' Special attributes: "__name__" is the class name; "__module__" ' - 'is\n' - ' the module name in which the class was defined; "__dict__" is ' - 'the\n' - ' dictionary containing the class?s namespace; "__bases__" is a ' - 'tuple\n' - ' containing the base classes, in the order of their occurrence ' - 'in\n' - ' the base class list; "__doc__" is the class?s documentation ' - 'string,\n' - ' or "None" if undefined; "__annotations__" (optional) is a\n' - ' dictionary containing *variable annotations* collected during ' - 'class\n' - ' body execution.\n' + ' Special attributes:\n' + '\n' + ' "__name__"\n' + ' The class name.\n' + '\n' + ' "__module__"\n' + ' The name of the module in which the class was defined.\n' + '\n' + ' "__dict__"\n' + ' The dictionary containing the class?s namespace.\n' + '\n' + ' "__bases__"\n' + ' A tuple containing the base classes, in the order of ' + 'their\n' + ' occurrence in the base class list.\n' + '\n' + ' "__doc__"\n' + ' The class?s documentation string, or "None" if undefined.\n' + '\n' + ' "__annotations__"\n' + ' A dictionary containing *variable annotations* collected\n' + ' during class body execution. For best practices on ' + 'working\n' + ' with "__annotations__", please see Annotations Best\n' + ' Practices.\n' '\n' 'Class instances\n' ' A class instance is created by calling a class object (see ' @@ -13072,6 +13272,7 @@ '\n' ' Special read-only attributes: "co_name" gives the function ' 'name;\n' + ' "co_qualname" gives the fully qualified function name;\n' ' "co_argcount" is the total number of positional arguments\n' ' (including positional-only arguments and arguments with ' 'default\n' @@ -13132,6 +13333,54 @@ ' "co_consts" is the documentation string of the function, or\n' ' "None" if undefined.\n' '\n' + ' codeobject.co_positions()\n' + '\n' + ' Returns an iterable over the source code positions of ' + 'each\n' + ' bytecode instruction in the code object.\n' + '\n' + ' The iterator returns tuples containing the "(start_line,\n' + ' end_line, start_column, end_column)". The *i-th* tuple\n' + ' corresponds to the position of the source code that ' + 'compiled\n' + ' to the *i-th* instruction. Column information is ' + '0-indexed\n' + ' utf-8 byte offsets on the given source line.\n' + '\n' + ' This positional information can be missing. A ' + 'non-exhaustive\n' + ' lists of cases where this may happen:\n' + '\n' + ' * Running the interpreter with "-X" "no_debug_ranges".\n' + '\n' + ' * Loading a pyc file compiled while using "-X"\n' + ' "no_debug_ranges".\n' + '\n' + ' * Position tuples corresponding to artificial ' + 'instructions.\n' + '\n' + ' * Line and column numbers that can?t be represented due ' + 'to\n' + ' implementation specific limitations.\n' + '\n' + ' When this occurs, some or all of the tuple elements can ' + 'be\n' + ' "None".\n' + '\n' + ' New in version 3.11.\n' + '\n' + ' Note:\n' + '\n' + ' This feature requires storing column positions in code\n' + ' objects which may result in a small increase of disk ' + 'usage\n' + ' of compiled Python files or interpreter memory usage. ' + 'To\n' + ' avoid storing the extra information and/or deactivate\n' + ' printing the extra traceback information, the "-X"\n' + ' "no_debug_ranges" command line flag or the\n' + ' "PYTHONNODEBUGRANGES" environment variable can be used.\n' + '\n' ' Frame objects\n' ' Frame objects represent execution frames. They may occur in\n' ' traceback objects (see below), and are also passed to ' @@ -13150,6 +13399,10 @@ ' gives the precise instruction (this is an index into the\n' ' bytecode string of the code object).\n' '\n' + ' Accessing "f_code" raises an auditing event ' + '"object.__getattr__"\n' + ' with arguments "obj" and ""f_code"".\n' + '\n' ' Special writable attributes: "f_trace", if not "None", is a\n' ' function called for various events during code execution ' '(this\n' @@ -13233,6 +13486,9 @@ ' the exception occurred in a "try" statement with no matching\n' ' except clause or with a finally clause.\n' '\n' + ' Accessing "tb_frame" raises an auditing event\n' + ' "object.__getattr__" with arguments "obj" and ""tb_frame"".\n' + '\n' ' Special writable attribute: "tb_next" is the next level in ' 'the\n' ' stack trace (towards the frame where the exception occurred), ' @@ -13283,9 +13539,8 @@ ' object actually returned is the wrapped object, which is not\n' ' subject to any further transformation. Static method objects ' 'are\n' - ' not themselves callable, although the objects they wrap ' - 'usually\n' - ' are. Static method objects are created by the built-in\n' + ' also callable. Static method objects are created by the ' + 'built-in\n' ' "staticmethod()" constructor.\n' '\n' ' Class method objects\n' @@ -14237,7 +14492,7 @@ '| | "s[i:i] = ' '[x]") | |\n' '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.pop([i])" | retrieves the item at *i* ' + '| "s.pop()" or "s.pop(i)" | retrieves the item at *i* ' 'and | (2) |\n' '| | also removes it from ' '*s* | |\n' @@ -14700,7 +14955,7 @@ '| | "s[i:i] = ' '[x]") | |\n' '+--------------------------------+----------------------------------+-----------------------+\n' - '| "s.pop([i])" | retrieves the item at ' + '| "s.pop()" or "s.pop(i)" | retrieves the item at ' '*i* and | (2) |\n' '| | also removes it from ' '*s* | |\n' @@ -14765,15 +15020,21 @@ ' u_expr ::= power | "-" u_expr | "+" u_expr | "~" u_expr\n' '\n' 'The unary "-" (minus) operator yields the negation of its numeric\n' - 'argument.\n' + 'argument; the operation can be overridden with the "__neg__()" ' + 'special\n' + 'method.\n' '\n' 'The unary "+" (plus) operator yields its numeric argument ' - 'unchanged.\n' + 'unchanged;\n' + 'the operation can be overridden with the "__pos__()" special ' + 'method.\n' '\n' 'The unary "~" (invert) operator yields the bitwise inversion of ' 'its\n' 'integer argument. The bitwise inversion of "x" is defined as\n' - '"-(x+1)". It only applies to integral numbers.\n' + '"-(x+1)". It only applies to integral numbers or to custom ' + 'objects\n' + 'that override the "__invert__()" special method.\n' '\n' 'In all three cases, if the argument does not have the proper type, ' 'a\n' diff --git a/Misc/NEWS.d/3.11.0a1.rst b/Misc/NEWS.d/3.11.0a1.rst new file mode 100644 index 0000000000000..ba07ef95b4801 --- /dev/null +++ b/Misc/NEWS.d/3.11.0a1.rst @@ -0,0 +1,5098 @@ +.. bpo: 42278 +.. date: 2021-08-29-12-39-44 +.. nonce: jvmQz_ +.. release date: 2021-10-05 +.. section: Security + +Replaced usage of :func:`tempfile.mktemp` with +:class:`~tempfile.TemporaryDirectory` to avoid a potential race condition. + +.. + +.. bpo: 44600 +.. date: 2021-07-25-20-04-54 +.. nonce: 0WMldg +.. section: Security + +Fix incorrect line numbers while tracing some failed patterns in :ref:`match +` statements. Patch by Charles Burkland. + +.. + +.. bpo: 41180 +.. date: 2021-06-29-23-40-22 +.. nonce: uTWHv_ +.. section: Security + +Add auditing events to the :mod:`marshal` module, and stop raising +``code.__init__`` events for every unmarshalled code object. Directly +instantiated code objects will continue to raise an event, and audit event +handlers should inspect or collect the raw marshal data. This reduces a +significant performance overhead when loading from ``.pyc`` files. + +.. + +.. bpo: 44394 +.. date: 2021-06-29-02-45-53 +.. nonce: A220N1 +.. section: Security + +Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the fix +for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy is most used +on Windows and macOS. + +.. + +.. bpo: 43124 +.. date: 2021-05-08-11-50-46 +.. nonce: 2CTM6M +.. section: Security + +Made the internal ``putcmd`` function in :mod:`smtplib` sanitize input for +presence of ``\r`` and ``\n`` characters to avoid (unlikely) command +injection. + +.. + +.. bpo: 44022 +.. date: 2021-05-05-17-37-04 +.. nonce: bS3XJ9 +.. section: Security + +:mod:`http.client` now avoids infinitely reading potential HTTP headers +after a ``100 Continue`` status response from the server. + +.. + +.. bpo: 43760 +.. date: 2021-10-04-16-11-50 +.. nonce: R9QoUv +.. section: Core and Builtins + +The number of hardware branches per instruction dispatch is reduced from two +to one by adding a special instruction for tracing. Patch by Mark Shannon. + +.. + +.. bpo: 45061 +.. date: 2021-09-21-22-27-25 +.. nonce: 5IOUf0 +.. section: Core and Builtins + +Add a deallocator to the bool type to detect refcount bugs in C extensions +which call Py_DECREF(Py_True) or Py_DECREF(Py_False) by mistake. Detect also +refcount bugs when the empty tuple singleton or the Unicode empty string +singleton is destroyed by mistake. Patch by Victor Stinner. + +.. + +.. bpo: 24076 +.. date: 2021-09-20-10-02-12 +.. nonce: ZFgFSj +.. section: Core and Builtins + +sum() was further optimised for summing up single digit integers. + +.. + +.. bpo: 45190 +.. date: 2021-09-14-10-02-12 +.. nonce: ZFRgSj +.. section: Core and Builtins + +Update Unicode databases to Unicode 14.0.0. + +.. + +.. bpo: 45167 +.. date: 2021-09-14-09-23-59 +.. nonce: CPSSoV +.. section: Core and Builtins + +Fix deepcopying of :class:`types.GenericAlias` objects. + +.. + +.. bpo: 45155 +.. date: 2021-09-09-15-05-17 +.. nonce: JRw9TG +.. section: Core and Builtins + +:meth:`int.to_bytes` and :meth:`int.from_bytes` now take a default value of +``"big"`` for the ``byteorder`` argument. :meth:`int.to_bytes` also takes a +default value of ``1`` for the ``length`` argument. + +.. + +.. bpo: 44219 +.. date: 2021-09-09-10-32-33 +.. nonce: WiYyjz +.. section: Core and Builtins + +Release the GIL while performing ``isatty`` system calls on arbitrary file +descriptors. In particular, this affects :func:`os.isatty`, +:func:`os.device_encoding` and :class:`io.TextIOWrapper`. By extension, +:func:`io.open` in text mode is also affected. This change solves a deadlock +in :func:`os.isatty`. Patch by Vincent Michel in :issue:`44219`. + +.. + +.. bpo: 44959 +.. date: 2021-09-08-08-29-41 +.. nonce: OSwwPf +.. section: Core and Builtins + +Added fallback to extension modules with '.sl' suffix on HP-UX + +.. + +.. bpo: 45121 +.. date: 2021-09-07-17-10-16 +.. nonce: iG-Hsf +.. section: Core and Builtins + +Fix issue where ``Protocol.__init__`` raises ``RecursionError`` when it's +called directly or via ``super()``. Patch provided by Yurii Karabas. + +.. + +.. bpo: 44348 +.. date: 2021-09-07-00-21-04 +.. nonce: f8w_Td +.. section: Core and Builtins + +The deallocator function of the :exc:`BaseException` type now uses the +trashcan mecanism to prevent stack overflow. For example, when a +:exc:`RecursionError` instance is raised, it can be linked to another +RecursionError through the ``__context__`` attribute or the +``__traceback__`` attribute, and then a chain of exceptions is created. When +the chain is destroyed, nested deallocator function calls can crash with a +stack overflow if the chain is too long compared to the available stack +memory. Patch by Victor Stinner. + +.. + +.. bpo: 45123 +.. date: 2021-09-06-21-52-45 +.. nonce: 8Eh9iI +.. section: Core and Builtins + +Fix PyAiter_Check to only check for the __anext__ presence (not for +__aiter__). Rename PyAiter_Check to PyAIter_Check, PyObject_GetAiter -> +PyObject_GetAIter. + +.. + +.. bpo: 1514420 +.. date: 2021-09-03-16-18-10 +.. nonce: 2Lumpj +.. section: Core and Builtins + +Interpreter no longer attempts to open files with names in angle brackets +(like "" or "") when formatting an exception. + +.. + +.. bpo: 41031 +.. date: 2021-09-03-12-35-17 +.. nonce: yPSJEs +.. section: Core and Builtins + +Match C and Python code formatting of unprintable exceptions and exceptions +in the :mod:`__main__` module. + +.. + +.. bpo: 37330 +.. date: 2021-09-02-01-28-01 +.. nonce: QDjM_l +.. section: Core and Builtins + +:func:`open`, :func:`io.open`, :func:`codecs.open` and +:class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") +in the file mode. This flag was deprecated since Python 3.3. Patch by Victor +Stinner. + +.. + +.. bpo: 45083 +.. date: 2021-09-01-23-55-49 +.. nonce: cLi9G3 +.. section: Core and Builtins + +When the interpreter renders an exception, its name now has a complete +qualname. Previously only the class name was concatenated to the module +name, which sometimes resulted in an incorrect full name being displayed. + +(This issue impacted only the C code exception rendering, the +:mod:`traceback` module was using qualname already). + +.. + +.. bpo: 34561 +.. date: 2021-09-01-19-21-48 +.. nonce: uMAVA- +.. section: Core and Builtins + +List sorting now uses the merge-ordering strategy from Munro and Wild's +``powersort()``. Unlike the former strategy, this is provably near-optimal +in the entropy of the distribution of run lengths. Most uses of +``list.sort()`` probably won't see a significant time difference, but may +see significant improvements in cases where the former strategy was +exceptionally poor. However, as these are all fast linear-time +approximations to a problem that's inherently at best quadratic-time to +solve truly optimally, it's also possible to contrive cases where the former +strategy did better. + +.. + +.. bpo: 45056 +.. date: 2021-09-01-16-55-43 +.. nonce: 7AK2d9 +.. section: Core and Builtins + +Compiler now removes trailing unused constants from co_consts. + +.. + +.. bpo: 45020 +.. date: 2021-08-31-17-44-51 +.. nonce: ZPI_3L +.. section: Core and Builtins + +Add a new command line option, "-X frozen_modules=[on|off]" to opt out of +(or into) using optional frozen modules. This defaults to "on" (or "off" if +it's a debug build). + +.. + +.. bpo: 45012 +.. date: 2021-08-31-11-09-52 +.. nonce: ueeOcx +.. section: Core and Builtins + +In :mod:`posix`, release GIL during ``stat()``, ``lstat()``, and +``fstatat()`` syscalls made by :func:`os.DirEntry.stat`. Patch by Stanis?aw +Skonieczny. + +.. + +.. bpo: 45018 +.. date: 2021-08-26-18-44-03 +.. nonce: pu8H9L +.. section: Core and Builtins + +Fixed pickling of range iterators that iterated for over ``2**32`` times. + +.. + +.. bpo: 45000 +.. date: 2021-08-25-23-17-32 +.. nonce: XjmyLl +.. section: Core and Builtins + +A :exc:`SyntaxError` is now raised when trying to delete :const:`__debug__`. +Patch by Dong-hee Na. + +.. + +.. bpo: 44963 +.. date: 2021-08-25-23-07-10 +.. nonce: 5EET8y +.. section: Core and Builtins + +Implement ``send()`` and ``throw()`` methods for ``anext_awaitable`` +objects. Patch by Pablo Galindo. + +.. + +.. bpo: 44962 +.. date: 2021-08-23-19-55-08 +.. nonce: J00ftt +.. section: Core and Builtins + +Fix a race in WeakKeyDictionary, WeakValueDictionary and WeakSet when two +threads attempt to commit the last pending removal. This fixes +asyncio.create_task and fixes a data loss in asyncio.run where +shutdown_asyncgens is not run + +.. + +.. bpo: 24234 +.. date: 2021-08-23-10-36-55 +.. nonce: MGVUQi +.. section: Core and Builtins + +Implement the :meth:`__bytes__` special method on the :class:`bytes` type, +so a bytes object ``b`` passes an ``isinstance(b, typing.SupportsBytes)`` +check. + +.. + +.. bpo: 24234 +.. date: 2021-08-22-12-28-50 +.. nonce: n3oTdx +.. section: Core and Builtins + +Implement the :meth:`__complex__` special method on the :class:`complex` +type, so a complex number ``z`` passes an ``isinstance(z, +typing.SupportsComplex)`` check. + +.. + +.. bpo: 44954 +.. date: 2021-08-19-14-43-24 +.. nonce: dLn3lg +.. section: Core and Builtins + +Fixed a corner case bug where the result of ``float.fromhex('0x.8p-1074')`` +was rounded the wrong way. + +.. + +.. bpo: 44947 +.. date: 2021-08-18-19-09-28 +.. nonce: mcvGdS +.. section: Core and Builtins + +Refine the syntax error for trailing commas in import statements. Patch by +Pablo Galindo. + +.. + +.. bpo: 44945 +.. date: 2021-08-18-11-14-38 +.. nonce: CO3s77 +.. section: Core and Builtins + +Specialize the BINARY_ADD instruction using the PEP 659 machinery. Adds five +new instructions: + +* BINARY_ADD_ADAPTIVE +* BINARY_ADD_FLOAT +* BINARY_ADD_INT +* BINARY_ADD_UNICODE +* BINARY_ADD_UNICODE_INPLACE_FAST + +.. + +.. bpo: 44929 +.. date: 2021-08-16-23-16-17 +.. nonce: qpMEky +.. section: Core and Builtins + +Fix some edge cases of ``enum.Flag`` string representation in the REPL. +Patch by Pablo Galindo. + +.. + +.. bpo: 44914 +.. date: 2021-08-16-11-36-02 +.. nonce: 6Lgrx3 +.. section: Core and Builtins + +Class version tags are no longer recycled. + +This means that a version tag serves as a unique identifier for the state of +a class. We rely on this for effective specialization of the LOAD_ATTR and +other instructions. + +.. + +.. bpo: 44698 +.. date: 2021-08-15-10-39-06 +.. nonce: lITKNc +.. section: Core and Builtins + +Restore behaviour of complex exponentiation with integer-valued exponent of +type :class:`float` or :class:`complex`. + +.. + +.. bpo: 44895 +.. date: 2021-08-14-20-13-21 +.. nonce: Ic9m90 +.. section: Core and Builtins + +A debug variable :envvar:`PYTHONDUMPREFSFILE` is added for creating a dump +file which is generated by :option:`--with-trace-refs`. Patch by Dong-hee +Na. + +.. + +.. bpo: 44900 +.. date: 2021-08-12-14-00-57 +.. nonce: w2gpwy +.. section: Core and Builtins + +Add five superinstructions for PEP 659 quickening: + +* LOAD_FAST LOAD_FAST +* STORE_FAST LOAD_FAST +* LOAD_FAST LOAD_CONST +* LOAD_CONST LOAD_FAST +* STORE_FAST STORE_FAST + +.. + +.. bpo: 44889 +.. date: 2021-08-11-20-45-02 +.. nonce: 2T3nTn +.. section: Core and Builtins + +Initial implementation of adaptive specialization of ``LOAD_METHOD``. The +following specialized forms were added: + +* ``LOAD_METHOD_CACHED`` + +* ``LOAD_METHOD_MODULE`` + +* ``LOAD_METHOD_CLASS`` + +.. + +.. bpo: 44890 +.. date: 2021-08-11-16-46-27 +.. nonce: PwNg8N +.. section: Core and Builtins + +Specialization stats are always collected in debug builds. + +.. + +.. bpo: 44885 +.. date: 2021-08-11-15-39-57 +.. nonce: i4noUO +.. section: Core and Builtins + +Correct the ast locations of f-strings with format specs and repeated +expressions. Patch by Pablo Galindo + +.. + +.. bpo: 44878 +.. date: 2021-08-11-14-12-41 +.. nonce: pAbBfc +.. section: Core and Builtins + +Remove the loop from the bytecode interpreter. All instructions end with a +DISPATCH macro, so the loop is now redundant. + +.. + +.. bpo: 44878 +.. date: 2021-08-11-12-03-52 +.. nonce: nEhjLi +.. section: Core and Builtins + +Remove switch statement for interpreter loop when using computed gotos. This +makes sure that we only have one dispatch table in the interpreter. + +.. + +.. bpo: 44874 +.. date: 2021-08-09-19-05-20 +.. nonce: oOcfU4 +.. section: Core and Builtins + +Deprecate the old trashcan macros +(``Py_TRASHCAN_SAFE_BEGIN``/``Py_TRASHCAN_SAFE_END``). They should be +replaced by the new macros ``Py_TRASHCAN_BEGIN`` and ``Py_TRASHCAN_END``. + +.. + +.. bpo: 44872 +.. date: 2021-08-09-16-16-03 +.. nonce: OKRlhK +.. section: Core and Builtins + +Use new trashcan macros (Py_TRASHCAN_BEGIN/END) in frameobject.c instead of +the old ones (Py_TRASHCAN_SAFE_BEGIN/END). + +.. + +.. bpo: 33930 +.. date: 2021-08-09-14-29-52 +.. nonce: --5LQ- +.. section: Core and Builtins + +Fix segmentation fault with deep recursion when cleaning method objects. +Patch by Augusto Goulart and Pablo Galindo. + +.. + +.. bpo: 25782 +.. date: 2021-08-07-21-39-19 +.. nonce: B22lMx +.. section: Core and Builtins + +Fix bug where ``PyErr_SetObject`` hangs when the current exception has a +cycle in its context chain. + +.. + +.. bpo: 44856 +.. date: 2021-08-07-01-26-12 +.. nonce: 9rk3li +.. section: Core and Builtins + +Fix reference leaks in the error paths of ``update_bases()`` and +``__build_class__``. Patch by Pablo Galindo. + +.. + +.. bpo: 44826 +.. date: 2021-08-05-17-49-55 +.. nonce: zQsyK5 +.. section: Core and Builtins + +Initial implementation of adaptive specialization of STORE_ATTR + +Three specialized forms of STORE_ATTR are added: + +* STORE_ATTR_SLOT + +* STORE_ATTR_SPLIT_KEYS + +* STORE_ATTR_WITH_HINT + +.. + +.. bpo: 44838 +.. date: 2021-08-05-17-42-03 +.. nonce: r_Lkj_ +.. section: Core and Builtins + +Fixed a bug that was causing the parser to raise an incorrect custom +:exc:`SyntaxError` for invalid 'if' expressions. Patch by Pablo Galindo. + +.. + +.. bpo: 44821 +.. date: 2021-08-04-11-37-38 +.. nonce: 67YHGI +.. section: Core and Builtins + +Create instance dictionaries (__dict__) eagerly, to improve regularity of +object layout and assist specialization. + +.. + +.. bpo: 44792 +.. date: 2021-07-31-12-12-57 +.. nonce: mOReTW +.. section: Core and Builtins + +Improve syntax errors for if expressions. Patch by Miguel Brito + +.. + +.. bpo: 34013 +.. date: 2021-07-27-11-14-22 +.. nonce: SjLFe- +.. section: Core and Builtins + +Generalize the invalid legacy statement custom error message (like the one +generated when "print" is called without parentheses) to include more +generic expressions. Patch by Pablo Galindo + +.. + +.. bpo: 44732 +.. date: 2021-07-26-15-27-03 +.. nonce: IxObt3 +.. section: Core and Builtins + +Rename ``types.Union`` to ``types.UnionType``. + +.. + +.. bpo: 44725 +.. date: 2021-07-23-15-17-01 +.. nonce: qcuKaa +.. section: Core and Builtins + +Expose specialization stats in python via +:func:`_opcode.get_specialization_stats`. + +.. + +.. bpo: 44717 +.. date: 2021-07-23-01-52-13 +.. nonce: -vVmAh +.. section: Core and Builtins + +Improve AttributeError on circular imports of submodules. + +.. + +.. bpo: 44698 +.. date: 2021-07-21-15-26-56 +.. nonce: DA4_0o +.. section: Core and Builtins + +Fix undefined behaviour in complex object exponentiation. + +.. + +.. bpo: 44653 +.. date: 2021-07-19-20-49-06 +.. nonce: WcqGyI +.. section: Core and Builtins + +Support :mod:`typing` types in parameter substitution in the union type. + +.. + +.. bpo: 44676 +.. date: 2021-07-19-19-53-46 +.. nonce: WgIMvh +.. section: Core and Builtins + +Add ability to serialise ``types.Union`` objects. Patch provided by Yurii +Karabas. + +.. + +.. bpo: 44633 +.. date: 2021-07-17-21-04-04 +.. nonce: 5-zKeI +.. section: Core and Builtins + +Parameter substitution of the union type with wrong types now raises +``TypeError`` instead of returning ``NotImplemented``. + +.. + +.. bpo: 44661 +.. date: 2021-07-17-14-20-59 +.. nonce: BQbXiH +.. section: Core and Builtins + +Update ``property_descr_set`` to use vectorcall if possible. Patch by +Dong-hee Na. + +.. + +.. bpo: 44662 +.. date: 2021-07-17-13-41-58 +.. nonce: q22kWR +.. section: Core and Builtins + +Add ``__module__`` to ``types.Union``. This also fixes ``types.Union`` +issues with ``typing.Annotated``. Patch provided by Yurii Karabas. + +.. + +.. bpo: 44655 +.. date: 2021-07-16-21-35-14 +.. nonce: 95I7M6 +.. section: Core and Builtins + +Include the name of the type in unset __slots__ attribute errors. Patch by +Pablo Galindo + +.. + +.. bpo: 44655 +.. date: 2021-07-16-20-25-37 +.. nonce: I3wRjL +.. section: Core and Builtins + +Don't include a missing attribute with the same name as the failing one when +offering suggestions for missing attributes. Patch by Pablo Galindo + +.. + +.. bpo: 44646 +.. date: 2021-07-16-09-59-13 +.. nonce: Yb6s05 +.. section: Core and Builtins + +Fix the hash of the union type: it no longer depends on the order of +arguments. + +.. + +.. bpo: 44636 +.. date: 2021-07-16-09-36-12 +.. nonce: ZWebi8 +.. section: Core and Builtins + +Collapse union of equal types. E.g. the result of ``int | int`` is now +``int``. Fix comparison of the union type with non-hashable objects. E.g. +``int | str == {}`` no longer raises a TypeError. + +.. + +.. bpo: 44611 +.. date: 2021-07-16-01-01-11 +.. nonce: LcfHN- +.. section: Core and Builtins + +On Windows, :func:`os.urandom`: uses BCryptGenRandom API instead of +CryptGenRandom API which is deprecated from Microsoft Windows API. Patch by +Dong-hee Na. + +.. + +.. bpo: 44635 +.. date: 2021-07-14-13-54-07 +.. nonce: 7ZMAdB +.. section: Core and Builtins + +Convert ``None`` to ``type(None)`` in the union type constructor. + +.. + +.. bpo: 26280 +.. date: 2021-07-14-10-31-10 +.. nonce: cgpM4B +.. section: Core and Builtins + +Implement adaptive specialization for BINARY_SUBSCR + +Three specialized forms of BINARY_SUBSCR are added: + +* BINARY_SUBSCR_LIST_INT + +* BINARY_SUBSCR_TUPLE_INT + +* BINARY_SUBSCR_DICT + +.. + +.. bpo: 44589 +.. date: 2021-07-13-23-19-41 +.. nonce: 59OH8T +.. section: Core and Builtins + +Mapping patterns in ``match`` statements with two or more equal literal keys +will now raise a :exc:`SyntaxError` at compile-time. + +.. + +.. bpo: 44606 +.. date: 2021-07-13-20-22-12 +.. nonce: S3Bv2w +.. section: Core and Builtins + +Fix ``__instancecheck__`` and ``__subclasscheck__`` for the union type. + +.. + +.. bpo: 42073 +.. date: 2021-07-13-17-47-32 +.. nonce: 9wopiC +.. section: Core and Builtins + +The ``@classmethod`` decorator can now wrap other classmethod-like +descriptors. + +.. + +.. bpo: 41972 +.. date: 2021-07-12-04-06-57 +.. nonce: nDX5k_ +.. section: Core and Builtins + +Tuned the string-searching algorithm of fastsearch.h to have a shorter inner +loop for most cases. + +.. + +.. bpo: 44590 +.. date: 2021-07-09-12-08-17 +.. nonce: a2ntVX +.. section: Core and Builtins + +All necessary data for executing a Python function (local variables, stack, +etc) is now kept in a per-thread stack. Frame objects are lazily allocated +on demand. This increases performance by about 7% on the standard benchmark +suite. Introspection and debugging are unaffected as frame objects are +always available when needed. Patch by Mark Shannon. + +.. + +.. bpo: 44584 +.. date: 2021-07-08-12-18-56 +.. nonce: qKnSqV +.. section: Core and Builtins + +The threading debug (:envvar:`PYTHONTHREADDEBUG` environment variable) is +deprecated in Python 3.10 and will be removed in Python 3.12. This feature +requires a debug build of Python. Patch by Victor Stinner. + +.. + +.. bpo: 43895 +.. date: 2021-07-07-16-05-35 +.. nonce: JFjR0- +.. section: Core and Builtins + +An obsolete internal cache of shared object file handles added in 1995 that +attempted, but did not guarantee, that a .so would not be dlopen'ed twice to +work around flaws in mid-1990s posix-ish operating systems has been removed +from dynload_shlib.c. + +.. + +.. bpo: 44490 +.. date: 2021-07-06-22-22-15 +.. nonce: BJxPbZ +.. section: Core and Builtins + +:mod:`typing` now searches for type parameters in ``types.Union`` objects. +``get_type_hints`` will also properly resolve annotations with nested +``types.Union`` objects. Patch provided by Yurii Karabas. + +.. + +.. bpo: 43950 +.. date: 2021-07-06-15-27-11 +.. nonce: LhL2-q +.. section: Core and Builtins + +Code objects can now provide the column information for instructions when +available. This is levaraged during traceback printing to show the +expressions responsible for errors. + +Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar as part of +:pep:`657`. + +.. + +.. bpo: 44562 +.. date: 2021-07-04-23-38-51 +.. nonce: QdeRQo +.. section: Core and Builtins + +Remove uses of :c:func:`PyObject_GC_Del` in error path when initializing +:class:`types.GenericAlias`. + +.. + +.. bpo: 41486 +.. date: 2021-07-04-17-41-47 +.. nonce: DiM24a +.. section: Core and Builtins + +Fix a memory consumption and copying performance regression in earlier 3.10 +beta releases if someone used an output buffer larger than 4GiB with +zlib.decompress on input data that expands that large. + +.. + +.. bpo: 43908 +.. date: 2021-07-03-00-20-39 +.. nonce: YHuV_s +.. section: Core and Builtins + +Heap types with the :const:`Py_TPFLAGS_IMMUTABLETYPE` flag can now inherit +the :pep:`590` vectorcall protocol. Previously, this was only possible for +:ref:`static types `. Patch by Erlend E. Aasland. + +.. + +.. bpo: 44553 +.. date: 2021-07-02-22-54-41 +.. nonce: l9YqGg +.. section: Core and Builtins + +Implement GC methods for ``types.Union`` to break reference cycles and +prevent memory leaks. + +.. + +.. bpo: 44490 +.. date: 2021-07-01-11-59-34 +.. nonce: xY80VR +.. section: Core and Builtins + +Add ``__parameters__`` attribute and ``__getitem__`` operator to +``types.Union``. Patch provided by Yurii Karabas. + +.. + +.. bpo: 44523 +.. date: 2021-06-29-11-49-29 +.. nonce: 67-ZIP +.. section: Core and Builtins + +Remove the pass-through for :func:`hash` of :class:`weakref.proxy` objects +to prevent unintended consequences when the original referred object dies +while the proxy is part of a hashable object. Patch by Pablo Galindo. + +.. + +.. bpo: 44483 +.. date: 2021-06-22-19-08-19 +.. nonce: eq2f7T +.. section: Core and Builtins + +Fix a crash in ``types.Union`` objects when creating a union of an object +with bad ``__module__`` field. + +.. + +.. bpo: 44486 +.. date: 2021-06-22-10-55-23 +.. nonce: wct-9X +.. section: Core and Builtins + +Modules will always have a dictionary, even when created by +``types.ModuleType.__new__()`` + +.. + +.. bpo: 44472 +.. date: 2021-06-21-11-19-54 +.. nonce: Vvm1yn +.. section: Core and Builtins + +Fix ltrace functionality when exceptions are raised. Patch by Pablo Galindo + +.. + +.. bpo: 12022 +.. date: 2021-06-20-10-53-21 +.. nonce: SW240M +.. section: Core and Builtins + +A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in +:keyword:`with` and :keyword:`async with` statements for objects which do +not support the :term:`context manager` or :term:`asynchronous context +manager` protocols correspondingly. + +.. + +.. bpo: 44297 +.. date: 2021-06-19-12-41-13 +.. nonce: F53vHj +.. section: Core and Builtins + +Make sure that the line number is set when entering a comprehension scope. +Ensures that backtraces inclusing generator expressions show the correct +line number. + +.. + +.. bpo: 44456 +.. date: 2021-06-18-22-08-25 +.. nonce: L0Rhko +.. section: Core and Builtins + +Improve the syntax error when mixing positional and keyword patterns. Patch +by Pablo Galindo. + +.. + +.. bpo: 44409 +.. date: 2021-06-13-23-12-18 +.. nonce: eW4LS- +.. section: Core and Builtins + +Fix error location information for tokenizer errors raised on initialization +of the tokenizer. Patch by Pablo Galindo. + +.. + +.. bpo: 44396 +.. date: 2021-06-11-18-17-42 +.. nonce: Z9EKim +.. section: Core and Builtins + +Fix a possible crash in the tokenizer when raising syntax errors for +unclosed strings. Patch by Pablo Galindo. + +.. + +.. bpo: 44376 +.. date: 2021-06-11-17-37-15 +.. nonce: zhM1UW +.. section: Core and Builtins + +Exact integer exponentiation (like ``i**2`` or ``pow(i, 2)``) with a small +exponent is much faster, due to reducing overhead in such cases. + +.. + +.. bpo: 44313 +.. date: 2021-06-10-16-10-39 +.. nonce: 34RjI8 +.. section: Core and Builtins + +Directly imported objects and modules (through import and from import +statements) don't generate ``LOAD_METHOD``/``CALL_METHOD`` for directly +accessed objects on their namespace. They now use the regular +``LOAD_ATTR``/``CALL_FUNCTION``. + +.. + +.. bpo: 44338 +.. date: 2021-06-10-10-06-18 +.. nonce: c4Myr4 +.. section: Core and Builtins + +Implement adaptive specialization for LOAD_GLOBAL + +Two specialized forms of LOAD_GLOBAL are added: + +* LOAD_GLOBAL_MODULE + +* LOAD_GLOBAL_BUILTIN + +.. + +.. bpo: 44368 +.. date: 2021-06-09-22-56-59 +.. nonce: vgT0Cx +.. section: Core and Builtins + +Improve syntax errors for invalid "as" targets. Patch by Pablo Galindo + +.. + +.. bpo: 44349 +.. date: 2021-06-08-22-49-06 +.. nonce: xgEgeA +.. section: Core and Builtins + +Fix an edge case when displaying text from files with encoding in syntax +errors. Patch by Pablo Galindo. + +.. + +.. bpo: 44337 +.. date: 2021-06-08-10-22-46 +.. nonce: RTjmIt +.. section: Core and Builtins + +Initial implementation of adaptive specialization of LOAD_ATTR + +Four specialized forms of LOAD_ATTR are added: + +* LOAD_ATTR_SLOT + +* LOAD_ATTR_SPLIT_KEYS + +* LOAD_ATTR_WITH_HINT + +* LOAD_ATTR_MODULE + +.. + +.. bpo: 44335 +.. date: 2021-06-08-01-13-47 +.. nonce: GQTTkl +.. section: Core and Builtins + +Fix a regression when identifying incorrect characters in syntax errors. +Patch by Pablo Galindo + +.. + +.. bpo: 43693 +.. date: 2021-06-07-15-13-44 +.. nonce: c_zDeY +.. section: Core and Builtins + +Computation of the offsets of cell variables is done in the compiler instead +of at runtime. This reduces the overhead of handling cell and free +variables, especially in the case where a variable is both an argument and +cell variable. + +.. + +.. bpo: 44317 +.. date: 2021-06-06-00-29-14 +.. nonce: xPPhcZ +.. section: Core and Builtins + +Improve tokenizer error with improved locations. Patch by Pablo Galindo. + +.. + +.. bpo: 44304 +.. date: 2021-06-05-02-34-57 +.. nonce: _MAoPc +.. section: Core and Builtins + +Fix a crash in the :mod:`sqlite3` module that happened when the garbage +collector clears :class:`sqlite.Statement` objects. Patch by Pablo Galindo + +.. + +.. bpo: 44305 +.. date: 2021-06-03-22-51-50 +.. nonce: 66dVDG +.. section: Core and Builtins + +Improve error message for ``try`` blocks without ``except`` or ``finally`` +blocks. Patch by Pablo Galindo. + +.. + +.. bpo: 43413 +.. date: 2021-05-30-16-37-47 +.. nonce: vYFPPC +.. section: Core and Builtins + +Constructors of subclasses of some buitin classes (e.g. :class:`tuple`, +:class:`list`, :class:`frozenset`) no longer accept arbitrary keyword +arguments. Subclass of :class:`set` can now define a ``__new__()`` method +with additional keyword parameters without overriding also ``__init__()``. + +.. + +.. bpo: 43667 +.. date: 2021-05-27-17-34-29 +.. nonce: ND9jP3 +.. section: Core and Builtins + +Improve Unicode support in non-UTF locales on Oracle Solaris. This issue +does not affect other Solaris systems. + +.. + +.. bpo: 43693 +.. date: 2021-05-26-19-10-47 +.. nonce: 1KSG9u +.. section: Core and Builtins + +A new opcode MAKE_CELL has been added that effectively moves some of the +work done on function entry into the compiler and into the eval loop. In +addition to creating the required cell objects, the new opcode converts +relevant arguments (and other locals) to cell variables on function entry. + +.. + +.. bpo: 44232 +.. date: 2021-05-25-18-20-10 +.. nonce: DMcCCf +.. section: Core and Builtins + +Fix a regression in :func:`type` when a metaclass raises an exception. The C +function :c:func:`type_new` must properly report the exception when a +metaclass constructor raises an exception and the winner class is not the +metaclass. Patch by Victor Stinner. + +.. + +.. bpo: 44201 +.. date: 2021-05-21-21-16-03 +.. nonce: bGaSjt +.. section: Core and Builtins + +Avoid side effects of checking for specialized syntax errors in the REPL +that was causing it to ask for extra tokens after a syntax error had been +detected. Patch by Pablo Galindo + +.. + +.. bpo: 43693 +.. date: 2021-05-21-20-53-49 +.. nonce: -NN3J_ +.. section: Core and Builtins + +``PyCodeObject`` gained ``co_fastlocalnames`` and ``co_fastlocalkinds`` as +the the authoritative source of fast locals info. Marshaled code objects +have changed accordingly. + +.. + +.. bpo: 44184 +.. date: 2021-05-21-01-42-45 +.. nonce: 9qOptC +.. section: Core and Builtins + +Fix a crash at Python exit when a deallocator function removes the last +strong reference to a heap type. Patch by Victor Stinner. + +.. + +.. bpo: 44187 +.. date: 2021-05-20-12-43-04 +.. nonce: 3lk0L1 +.. section: Core and Builtins + +Implement quickening in the interpreter. This offers no advantages as yet, +but is an enabler of future optimizations. See PEP 659 for full explanation. + +.. + +.. bpo: 44180 +.. date: 2021-05-19-20-33-36 +.. nonce: mQVaAs +.. section: Core and Builtins + +The parser doesn't report generic syntax errors that happen in a position +further away that the one it reached in the first pass. Patch by Pablo +Galindo + +.. + +.. bpo: 44168 +.. date: 2021-05-18-11-27-02 +.. nonce: mgB-rt +.. section: Core and Builtins + +Fix error message in the parser involving keyword arguments with invalid +expressions. Patch by Pablo Galindo + +.. + +.. bpo: 44156 +.. date: 2021-05-17-20-44-45 +.. nonce: 8KSp9l +.. section: Core and Builtins + +String caches in ``compile.c`` are now subinterpreter compatible. + +.. + +.. bpo: 44143 +.. date: 2021-05-15-17-30-57 +.. nonce: 7UTS6H +.. section: Core and Builtins + +Fixed a crash in the parser that manifest when raising tokenizer errors when +an existing exception was present. Patch by Pablo Galindo. + +.. + +.. bpo: 44032 +.. date: 2021-05-14-20-03-32 +.. nonce: OzT1ob +.. section: Core and Builtins + +Move 'fast' locals and other variables from the frame object to a per-thread +datastack. + +.. + +.. bpo: 44114 +.. date: 2021-05-12-14-26-16 +.. nonce: p-WfAE +.. section: Core and Builtins + +Fix incorrect dictkeys_reversed and dictitems_reversed function signatures +in C code, which broke webassembly builds. + +.. + +.. bpo: 44110 +.. date: 2021-05-11-21-52-44 +.. nonce: VqbAsB +.. section: Core and Builtins + +Improve :func:`str.__getitem__` error message + +.. + +.. bpo: 26110 +.. date: 2021-05-10-18-49-13 +.. nonce: EQzqqA +.. section: Core and Builtins + +Add ``CALL_METHOD_KW`` opcode to speed up method calls with keyword +arguments. Idea originated from PyPy. A side effect is executing +``CALL_METHOD`` is now branchless in the evaluation loop. + +.. + +.. bpo: 28307 +.. date: 2021-05-08-19-54-57 +.. nonce: 7ysaVW +.. section: Core and Builtins + +Compiler now optimizes simple C-style formatting with literal format +containing only format codes %s, %r and %a by converting them to f-string +expressions. + +.. + +.. bpo: 43149 +.. date: 2021-05-08-17-18-37 +.. nonce: Kp5FxD +.. section: Core and Builtins + +Corrent the syntax error message regarding multiple exception types to not +refer to "exception groups". Patch by Pablo Galindo + +.. + +.. bpo: 43822 +.. date: 2021-05-04-01-01-04 +.. nonce: 9VeCg0 +.. section: Core and Builtins + +The parser will prioritize tokenizer errors over custom syntax errors when +raising exceptions. Patch by Pablo Galindo. + +.. + +.. bpo: 40222 +.. date: 2021-04-30-15-48-36 +.. nonce: j3VxeX +.. section: Core and Builtins + +"Zero cost" exception handling. + +* Uses a lookup table to determine how to handle exceptions. +* Removes SETUP_FINALLY and POP_TOP block instructions, eliminating the runtime overhead of try statements. +* Reduces the size of the frame object by about 60%. + +Patch by Mark Shannon + +.. + +.. bpo: 43918 +.. date: 2021-04-23-03-46-45 +.. nonce: nNDY3S +.. section: Core and Builtins + +Document the signature and ``default`` argument in the docstring of the new +``anext`` builtin. + +.. + +.. bpo: 43833 +.. date: 2021-04-18-18-07-33 +.. nonce: oChkCi +.. section: Core and Builtins + +Emit a deprecation warning if the numeric literal is immediately followed by +one of keywords: and, else, for, if, in, is, or. Raise a syntax error with +more informative message if it is immediately followed by other keyword or +identifier. + +.. + +.. bpo: 43879 +.. date: 2021-04-17-16-08-00 +.. nonce: zkyJgh +.. section: Core and Builtins + +Add native_thread_id to PyThreadState. Patch by Gabriele N. Tornetta. + +.. + +.. bpo: 43693 +.. date: 2021-04-02-15-02-16 +.. nonce: l3Ureu +.. section: Core and Builtins + +Compute cell offsets relative to locals in compiler. Allows the interpreter +to treats locals and cells a single array, which is slightly more efficient. +Also make the LOAD_CLOSURE opcode an alias for LOAD_FAST. Preserving +LOAD_CLOSURE helps keep bytecode a bit more readable. + +.. + +.. bpo: 17792 +.. date: 2021-03-22-17-50-30 +.. nonce: _zssjS +.. section: Core and Builtins + +More accurate error messages for access of unbound locals or free vars. + +.. + +.. bpo: 28146 +.. date: 2021-01-13-19-34-41 +.. nonce: AZBBkH +.. section: Core and Builtins + +Fix a confusing error message in :func:`str.format`. + +.. + +.. bpo: 11105 +.. date: 2020-06-02-13-21-14 +.. nonce: wceryW +.. section: Core and Builtins + +When compiling :class:`ast.AST` objects with recursive references through +:func:`compile`, the interpreter doesn't crash anymore instead it raises a +:exc:`RecursionError`. + +.. + +.. bpo: 39091 +.. date: 2019-12-21-14-18-32 +.. nonce: dOexgQ +.. section: Core and Builtins + +Fix crash when using passing a non-exception to a generator's ``throw()`` +method. Patch by Noah Oxer + +.. + +.. bpo: 33346 +.. date: 2018-05-11-12-44-03 +.. nonce: ZgBkvB +.. section: Core and Builtins + +Asynchronous comprehensions are now allowed inside comprehensions in +asynchronous functions. Outer comprehensions implicitly become +asynchronous. + +.. + +.. bpo: 45371 +.. date: 2021-10-05-11-03-48 +.. nonce: NOwcDJ +.. section: Library + +Fix clang rpath issue in :mod:`distutils`. The UnixCCompiler now uses +correct clang option to add a runtime library directory (rpath) to a shared +library. + +.. + +.. bpo: 45329 +.. date: 2021-10-01-13-09-53 +.. nonce: 9iMYaO +.. section: Library + +Fix freed memory access in :class:`pyexpat.xmlparser` when building it with +an installed expat library <= 2.2.0. + +.. + +.. bpo: 41710 +.. date: 2021-09-30-23-00-18 +.. nonce: svuloZ +.. section: Library + +On Unix, if the ``sem_clockwait()`` function is available in the C library +(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses +the monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather +than using the system clock (:data:`time.CLOCK_REALTIME`), to not be +affected by system clock changes. Patch by Victor Stinner. + +.. + +.. bpo: 1596321 +.. date: 2021-09-24-17-20-23 +.. nonce: 3nhPUk +.. section: Library + +Fix the :func:`threading._shutdown` function when the :mod:`threading` +module was imported first from a thread different than the main thread: no +longer log an error at Python exit. + +.. + +.. bpo: 45274 +.. date: 2021-09-23-22-17-26 +.. nonce: gPpa4E +.. section: Library + +Fix a race condition in the :meth:`Thread.join() ` +method of the :mod:`threading` module. If the function is interrupted by a +signal and the signal handler raises an exception, make sure that the thread +remains in a consistent state to prevent a deadlock. Patch by Victor +Stinner. + +.. + +.. bpo: 21302 +.. date: 2021-09-22-23-56-15 +.. nonce: vvQ3Su +.. section: Library + +In Unix operating systems, :func:`time.sleep` now uses the ``nanosleep()`` +function, if ``clock_nanosleep()`` is not available but ``nanosleep()`` is +available. ``nanosleep()`` allows to sleep with nanosecond precision. + +.. + +.. bpo: 21302 +.. date: 2021-09-20-22-46-40 +.. nonce: h56430 +.. section: Library + +On Windows, :func:`time.sleep` now uses a waitable timer which has a +resolution of 100 nanoseconds (10\ :sup:`-7` seconds). Previously, it had a +resolution of 1 millisecond (10\ :sup:`-3` seconds). Patch by Livius and +Victor Stinner. + +.. + +.. bpo: 45238 +.. date: 2021-09-18-16-56-33 +.. nonce: Hng_9V +.. section: Library + +Fix :meth:`unittest.IsolatedAsyncioTestCase.debug`: it runs now asynchronous +methods and callbacks. + +.. + +.. bpo: 36674 +.. date: 2021-09-18-13-14-57 +.. nonce: a2k5Zb +.. section: Library + +:meth:`unittest.TestCase.debug` raises now a :class:`unittest.SkipTest` if +the class or the test method are decorated with the skipping decorator. + +.. + +.. bpo: 45235 +.. date: 2021-09-17-16-55-37 +.. nonce: sXnmPA +.. section: Library + +Fix an issue where argparse would not preserve values in a provided +namespace when using a subparser with defaults. + +.. + +.. bpo: 45183 +.. date: 2021-09-17-15-58-53 +.. nonce: Vv_vch +.. section: Library + +Have zipimport.zipimporter.find_spec() not raise an exception when the +underlying zip file has been deleted and the internal cache has been reset +via invalidate_cache(). + +.. + +.. bpo: 45234 +.. date: 2021-09-17-11-20-55 +.. nonce: qUcTVt +.. section: Library + +Fixed a regression in :func:`~shutil.copyfile`, :func:`~shutil.copy`, +:func:`~shutil.copy2` raising :exc:`FileNotFoundError` when source is a +directory, which should raise :exc:`IsADirectoryError` + +.. + +.. bpo: 45228 +.. date: 2021-09-17-09-59-33 +.. nonce: WV1dcT +.. section: Library + +Fix stack buffer overflow in parsing J1939 network address. + +.. + +.. bpo: 45225 +.. date: 2021-09-16-19-02-14 +.. nonce: xmKV4i +.. section: Library + +use map function instead of genexpr in capwords. + +.. + +.. bpo: 42135 +.. date: 2021-09-13-19-32-58 +.. nonce: 1ZAHqR +.. section: Library + +Fix typo: ``importlib.find_loader`` is really slated for removal in Python +3.12 not 3.10, like the others in GH-25169. + +Patch by Hugo van Kemenade. + +.. + +.. bpo: 20524 +.. date: 2021-09-13-14-59-01 +.. nonce: PMQ1Fh +.. section: Library + +Improves error messages on ``.format()`` operation for ``str``, ``float``, +``int``, and ``complex``. New format now shows the problematic pattern and +the object type. + +.. + +.. bpo: 45168 +.. date: 2021-09-13-14-28-49 +.. nonce: Z1mfW4 +.. section: Library + +Change :func:`dis.dis` output to omit op arg values that cannot be resolved +due to ``co_consts``, ``co_names`` etc not being provided. Previously the +oparg itself was repeated in the value field, which is not useful and can be +confusing. + +.. + +.. bpo: 21302 +.. date: 2021-09-11-18-44-40 +.. nonce: QxHRpR +.. section: Library + +In Unix operating systems, :func:`time.sleep` now uses the +``clock_nanosleep()`` function, if available, which allows to sleep for an +interval specified with nanosecond precision. + +.. + +.. bpo: 45173 +.. date: 2021-09-11-17-46-20 +.. nonce: UptGAn +.. section: Library + +Remove from the :mod:`configparser` module: the :class:`SafeConfigParser` +class, the :attr:`filename` property of the :class:`ParsingError` class, the +:meth:`readfp` method of the :class:`ConfigParser` class, deprecated since +Python 3.2. + +Patch by Hugo van Kemenade. + +.. + +.. bpo: 44987 +.. date: 2021-09-11-14-41-02 +.. nonce: Mt8DiX +.. section: Library + +Pure ASCII strings are now normalized in constant time by +:func:`unicodedata.normalize`. Patch by Dong-hee Na. + +.. + +.. bpo: 35474 +.. date: 2021-09-11-10-45-12 +.. nonce: tEY3SD +.. section: Library + +Calling :func:`mimetypes.guess_all_extensions` with ``strict=False`` no +longer affects the result of the following call with ``strict=True``. Also, +mutating the returned list no longer affects the global state. + +.. + +.. bpo: 45166 +.. date: 2021-09-10-21-35-53 +.. nonce: UHipXF +.. section: Library + +:func:`typing.get_type_hints` now works with :data:`~typing.Final` wrapped +in :class:`~typing.ForwardRef`. + +.. + +.. bpo: 45162 +.. date: 2021-09-10-13-20-53 +.. nonce: 2Jh-lq +.. section: Library + +Remove many old deprecated :mod:`unittest` features: + +* "``fail*``" and "``assert*``" aliases of :class:`~unittest.TestCase` methods. +* Broken from start :class:`~unittest.TestCase` method ``assertDictContainsSubset()``. +* Ignored :meth:` TestLoader.loadTestsFromModule` parameter *use_load_tests*. +* Old alias ``_TextTestResult`` of :class:`~unittest.TextTestResult`. + +.. + +.. bpo: 38371 +.. date: 2021-09-08-13-19-29 +.. nonce: y1kEfP +.. section: Library + +Remove the deprecated ``split()`` method of :class:`_tkinter.TkappType`. +Patch by Erlend E. Aasland. + +.. + +.. bpo: 20499 +.. date: 2021-09-08-01-19-31 +.. nonce: tSxx8Y +.. section: Library + +Improve the speed and accuracy of statistics.pvariance(). + +.. + +.. bpo: 45132 +.. date: 2021-09-07-16-33-51 +.. nonce: WI9zQY +.. section: Library + +Remove :meth:`__getitem__` methods of +:class:`xml.dom.pulldom.DOMEventStream`, :class:`wsgiref.util.FileWrapper` +and :class:`fileinput.FileInput`, deprecated since Python 3.9. + +Patch by Hugo van Kemenade. + +.. + +.. bpo: 45129 +.. date: 2021-09-07-14-27-39 +.. nonce: vXH0gw +.. section: Library + +Due to significant security concerns, the *reuse_address* parameter of +:meth:`asyncio.loop.create_datagram_endpoint`, disabled in Python 3.9, is +now entirely removed. This is because of the behavior of the socket option +``SO_REUSEADDR`` in UDP. + +Patch by Hugo van Kemenade. + +.. + +.. bpo: 45124 +.. date: 2021-09-07-09-13-27 +.. nonce: Kw5AUs +.. section: Library + +The ``bdist_msi`` command, deprecated in Python 3.9, is now removed. + +Use ``bdist_wheel`` (wheel packages) instead. + +Patch by Hugo van Kemenade. + +.. + +.. bpo: 30856 +.. date: 2021-09-05-21-37-28 +.. nonce: jj86y0 +.. section: Library + +:class:`unittest.TestResult` methods +:meth:`~unittest.TestResult.addFailure`, +:meth:`~unittest.TestResult.addError`, :meth:`~unittest.TestResult.addSkip` +and :meth:`~unittest.TestResult.addSubTest` are now called immediately after +raising an exception in test or finishing a subtest. Previously they were +called only after finishing the test clean up. + +.. + +.. bpo: 45034 +.. date: 2021-09-05-20-33-25 +.. nonce: 62NLD5 +.. section: Library + +Changes how error is formatted for ``struct.pack`` with ``'H'`` and ``'h'`` +modes and too large / small numbers. Now it shows the actual numeric limits, +while previously it was showing arithmetic expressions. + +.. + +.. bpo: 25894 +.. date: 2021-09-05-13-15-08 +.. nonce: zjbi2f +.. section: Library + +:mod:`unittest` now always reports skipped and failed subtests separately: +separate characters in default mode and separate lines in verbose mode. Also +the test description is now output for errors in test method, class and +module cleanups. + +.. + +.. bpo: 45081 +.. date: 2021-09-02-12-42-25 +.. nonce: tOjJ1k +.. section: Library + +Fix issue when dataclasses that inherit from ``typing.Protocol`` subclasses +have wrong ``__init__``. Patch provided by Yurii Karabas. + +.. + +.. bpo: 45085 +.. date: 2021-09-02-00-47-14 +.. nonce: mMnaDv +.. section: Library + +The ``binhex`` module, deprecated in Python 3.9, is now removed. The +following :mod:`binascii` functions, deprecated in Python 3.9, are now also +removed: + +* ``a2b_hqx()``, ``b2a_hqx()``; +* ``rlecode_hqx()``, ``rledecode_hqx()``. + +The :func:`binascii.crc_hqx` function remains available. + +Patch by Victor Stinner. + +.. + +.. bpo: 40360 +.. date: 2021-09-02-00-18-32 +.. nonce: 9nmMtB +.. section: Library + +The :mod:`lib2to3` package is now deprecated and may not be able to parse +Python 3.10 or newer. See the :pep:`617` (New PEG parser for CPython). Patch +by Victor Stinner. + +.. + +.. bpo: 45075 +.. date: 2021-09-01-15-27-00 +.. nonce: 9xUpvt +.. section: Library + +Rename :meth:`traceback.StackSummary.format_frame` to +:meth:`traceback.StackSummary.format_frame_summary`. This method was added +for 3.11 so it was not released yet. + +Updated code and docs to better distinguish frame and FrameSummary. + +.. + +.. bpo: 31299 +.. date: 2021-08-30-13-55-09 +.. nonce: 9QzjZs +.. section: Library + +Add option to completely drop frames from a traceback by returning ``None`` +from a :meth:`~traceback.StackSummary.format_frame` override. + +.. + +.. bpo: 41620 +.. date: 2021-08-29-14-49-22 +.. nonce: WJ6PFL +.. section: Library + +:meth:`~unittest.TestCase.run` now always return a +:class:`~unittest.TestResult` instance. Previously it returned ``None`` if +the test class or method was decorated with a skipping decorator. + +.. + +.. bpo: 45021 +.. date: 2021-08-28-13-00-12 +.. nonce: rReeaj +.. section: Library + +Fix a potential deadlock at shutdown of forked children when using +:mod:`concurrent.futures` module + +.. + +.. bpo: 43913 +.. date: 2021-08-27-23-40-51 +.. nonce: Uo1Gt5 +.. section: Library + +Fix bugs in cleaning up classes and modules in :mod:`unittest`: + +* Functions registered with :func:`~unittest.addModuleCleanup` were not called unless the user defines ``tearDownModule()`` in their test module. +* Functions registered with :meth:`~unittest.TestCase.addClassCleanup` were not called if ``tearDownClass`` is set to ``None``. +* Buffering in :class:`~unittest.TestResult` did not work with functions registered with ``addClassCleanup()`` and ``addModuleCleanup()``. +* Errors in functions registered with ``addClassCleanup()`` and ``addModuleCleanup()`` were not handled correctly in buffered and debug modes. +* Errors in ``setUpModule()`` and functions registered with ``addModuleCleanup()`` were reported in wrong order. +* And several lesser bugs. + +.. + +.. bpo: 45030 +.. date: 2021-08-27-19-01-23 +.. nonce: tAmBbY +.. section: Library + +Fix integer overflow in pickling and copying the range iterator. + +.. + +.. bpo: 45001 +.. date: 2021-08-26-16-25-48 +.. nonce: tn_dKp +.. section: Library + +Made email date parsing more robust against malformed input, namely a +whitespace-only ``Date:`` header. Patch by Wouter Bolsterlee. + +.. + +.. bpo: 45010 +.. date: 2021-08-26-09-54-14 +.. nonce: Cn23bQ +.. section: Library + +Remove support of special method ``__div__`` in :mod:`unittest.mock`. It is +not used in Python 3. + +.. + +.. bpo: 39218 +.. date: 2021-08-25-20-18-31 +.. nonce: BlO6jW +.. section: Library + +Improve accuracy of variance calculations by using ``x*x`` instead of +``x**2``. + +.. + +.. bpo: 43613 +.. date: 2021-08-25-10-28-49 +.. nonce: WkYmI0 +.. section: Library + +Improve the speed of :func:`gzip.compress` and :func:`gzip.decompress` by +compressing and decompressing at once in memory instead of in a streamed +fashion. + +.. + +.. bpo: 37596 +.. date: 2021-08-23-21-39-59 +.. nonce: ojRcwB +.. section: Library + +Ensure that :class:`set` and :class:`frozenset` objects are always +:mod:`marshalled ` reproducibly. + +.. + +.. bpo: 44019 +.. date: 2021-08-22-13-25-17 +.. nonce: BN8HDy +.. section: Library + +A new function ``operator.call`` has been added, such that +``operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)``. + +.. + +.. bpo: 42255 +.. date: 2021-08-19-23-49-10 +.. nonce: ofe3ms +.. section: Library + +:class:`webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. +It is untested and undocumented and also not used by webbrowser itself. +Patch by Dong-hee Na. + +.. + +.. bpo: 44955 +.. date: 2021-08-19-15-03-54 +.. nonce: 1mxFQS +.. section: Library + +Method :meth:`~unittest.TestResult.stopTestRun` is now always called in pair +with method :meth:`~unittest.TestResult.startTestRun` for +:class:`~unittest.TestResult` objects implicitly created in +:meth:`~unittest.TestCase.run`. Previously it was not called for test +methods and classes decorated with a skipping decorator. + +.. + +.. bpo: 39039 +.. date: 2021-08-18-10-36-14 +.. nonce: A63LYh +.. section: Library + +tarfile.open raises :exc:`~tarfile.ReadError` when a zlib error occurs +during file extraction. + +.. + +.. bpo: 44935 +.. date: 2021-08-17-16-01-44 +.. nonce: roUl0G +.. section: Library + +:mod:`subprocess` on Solaris now also uses :func:`os.posix_spawn()` for +better performance. + +.. + +.. bpo: 44911 +.. date: 2021-08-14-00-55-16 +.. nonce: uk3hYk +.. section: Library + +:class:`~unittest.IsolatedAsyncioTestCase` will no longer throw an exception +while cancelling leaked tasks. Patch by Bar Harel. + +.. + +.. bpo: 41322 +.. date: 2021-08-12-16-22-16 +.. nonce: utscTd +.. section: Library + +Added ``DeprecationWarning`` for tests and async tests that return a +value!=None (as this may indicate an improperly written test, for example a +test written as a generator function). + +.. + +.. bpo: 44524 +.. date: 2021-08-10-16-57-10 +.. nonce: dk9QX4 +.. section: Library + +Make exception message more useful when subclass from typing special form +alias. Patch provided by Yurii Karabas. + +.. + +.. bpo: 38956 +.. date: 2021-08-09-13-17-10 +.. nonce: owWLNv +.. section: Library + +:class:`argparse.BooleanOptionalAction`'s default value is no longer printed +twice when used with :class:`argparse.ArgumentDefaultsHelpFormatter`. + +.. + +.. bpo: 44860 +.. date: 2021-08-07-22-51-32 +.. nonce: PTvRrU +.. section: Library + +Fix the ``posix_user`` scheme in :mod:`sysconfig` to not depend on +:data:`sys.platlibdir`. + +.. + +.. bpo: 44859 +.. date: 2021-08-07-17-28-56 +.. nonce: CCopjk +.. section: Library + +Improve error handling in :mod:`sqlite3` and raise more accurate exceptions. + +* :exc:`MemoryError` is now raised instead of :exc:`sqlite3.Warning` when memory is not enough for encoding a statement to UTF-8 in ``Connection.__call__()`` and ``Cursor.execute()``. +* :exc:`UnicodEncodeError` is now raised instead of :exc:`sqlite3.Warning` when the statement contains surrogate characters in ``Connection.__call__()`` and ``Cursor.execute()``. +* :exc:`TypeError` is now raised instead of :exc:`ValueError` for non-string script argument in ``Cursor.executescript()``. +* :exc:`ValueError` is now raised for script containing the null character instead of truncating it in ``Cursor.executescript()``. +* Correctly handle exceptions raised when getting boolean value of the result of the progress handler. +* Add many tests covering different corner cases. + +.. + +.. bpo: 44581 +.. date: 2021-08-06-19-15-52 +.. nonce: oFDBTB +.. section: Library + +Upgrade bundled pip to 21.2.3 and setuptools to 57.4.0 + +.. + +.. bpo: 44849 +.. date: 2021-08-06-13-00-28 +.. nonce: O78F_f +.. section: Library + +Fix the :func:`os.set_inheritable` function on FreeBSD 14 for file +descriptor opened with the :data:`~os.O_PATH` flag: ignore the +:data:`~errno.EBADF` error on ``ioctl()``, fallback on the ``fcntl()`` +implementation. Patch by Victor Stinner. + +.. + +.. bpo: 44605 +.. date: 2021-08-06-09-43-50 +.. nonce: q4YSBZ +.. section: Library + +The @functools.total_ordering() decorator now works with metaclasses. + +.. + +.. bpo: 44524 +.. date: 2021-08-05-18-20-17 +.. nonce: 9T1tfe +.. section: Library + +Fixed an issue wherein the ``__name__`` and ``__qualname__`` attributes of +subscribed specialforms could be ``None``. + +.. + +.. bpo: 44839 +.. date: 2021-08-05-14-59-39 +.. nonce: MURNL9 +.. section: Library + +:class:`MemoryError` raised in user-defined functions will now produce a +``MemoryError`` in :mod:`sqlite3`. :class:`OverflowError` will now be +converted to :class:`~sqlite3.DataError`. Previously +:class:`~sqlite3.OperationalError` was produced in these cases. + +.. + +.. bpo: 44822 +.. date: 2021-08-04-12-29-00 +.. nonce: zePNXA +.. section: Library + +:mod:`sqlite3` user-defined functions and aggregators returning +:class:`strings ` with embedded NUL characters are no longer truncated. +Patch by Erlend E. Aasland. + +.. + +.. bpo: 44801 +.. date: 2021-08-03-20-37-45 +.. nonce: i49Aug +.. section: Library + +Ensure that the :class:`~typing.ParamSpec` variable in Callable can only be +substituted with a parameters expression (a list of types, an ellipsis, +ParamSpec or Concatenate). + +.. + +.. bpo: 44806 +.. date: 2021-08-02-14-37-32 +.. nonce: wOW_Qn +.. section: Library + +Non-protocol subclasses of :class:`typing.Protocol` ignore now the +``__init__`` method inherited from protocol base classes. + +.. + +.. bpo: 27275 +.. date: 2021-08-01-19-49-09 +.. nonce: QsvE0k +.. section: Library + +:meth:`collections.OrderedDict.popitem` and +:meth:`collections.OrderedDict.pop` no longer call ``__getitem__`` and +``__delitem__`` methods of the OrderedDict subclasses. + +.. + +.. bpo: 44793 +.. date: 2021-07-31-20-28-20 +.. nonce: woaQSg +.. section: Library + +Fix checking the number of arguments when subscribe a generic type with +``ParamSpec`` parameter. + +.. + +.. bpo: 44784 +.. date: 2021-07-31-08-45-31 +.. nonce: fIMIDS +.. section: Library + +In importlib.metadata tests, override warnings behavior under expected +DeprecationWarnings (importlib_metadata 4.6.3). + +.. + +.. bpo: 44667 +.. date: 2021-07-30-23-27-30 +.. nonce: tu0Xrv +.. section: Library + +The :func:`tokenize.tokenize` doesn't incorrectly generate a ``NEWLINE`` +token if the source doesn't end with a new line character but the last line +is a comment, as the function is already generating a ``NL`` token. Patch by +Pablo Galindo + +.. + +.. bpo: 44771 +.. date: 2021-07-28-22-53-18 +.. nonce: BvLdnU +.. section: Library + +Added ``importlib.simple`` module implementing adapters from a low-level +resources reader interface to a ``TraversableResources`` interface. Legacy +API (``path``, ``contents``, ...) is now supported entirely by the +``.files()`` API with a compatibility shim supplied for resource loaders +without that functionality. Feature parity with ``importlib_resources`` 5.2. + +.. + +.. bpo: 44752 +.. date: 2021-07-27-22-11-29 +.. nonce: _bvbrZ +.. section: Library + +:mod:`rcompleter` does not call :func:`getattr` on :class:`property` objects +to avoid the side-effect of evaluating the corresponding method. + +.. + +.. bpo: 44747 +.. date: 2021-07-27-12-06-19 +.. nonce: epUzZz +.. section: Library + +Refactor usage of ``sys._getframe`` in ``typing`` module. Patch provided by +Yurii Karabas. + +.. + +.. bpo: 42378 +.. date: 2021-07-25-08-17-55 +.. nonce: WIhUZK +.. section: Library + +Fixes the issue with log file being overwritten when +:class:`logging.FileHandler` is used in :mod:`atexit` with *filemode* set to +``'w'``. Note this will cause the message in *atexit* not being logged if +the log stream is already closed due to shutdown of logging. + +.. + +.. bpo: 44720 +.. date: 2021-07-24-02-17-59 +.. nonce: shU5Qm +.. section: Library + +``weakref.proxy`` objects referencing non-iterators now raise ``TypeError`` +rather than dereferencing the null ``tp_iternext`` slot and crashing. + +.. + +.. bpo: 44704 +.. date: 2021-07-21-23-16-30 +.. nonce: iqHLxQ +.. section: Library + +The implementation of ``collections.abc.Set._hash()`` now matches that of +``frozenset.__hash__()``. + +.. + +.. bpo: 44666 +.. date: 2021-07-21-10-43-22 +.. nonce: CEThkv +.. section: Library + +Fixed issue in :func:`compileall.compile_file` when ``sys.stdout`` is +redirected. Patch by Stefan H?lzl. + +.. + +.. bpo: 44688 +.. date: 2021-07-20-23-28-26 +.. nonce: buFgz3 +.. section: Library + +:meth:`sqlite3.Connection.create_collation` now accepts non-ASCII collation +names. Patch by Erlend E. Aasland. + +.. + +.. bpo: 44690 +.. date: 2021-07-20-22-03-24 +.. nonce: tV7Zjg +.. section: Library + +Adopt *binacii.a2b_base64*'s strict mode in *base64.b64decode*. + +.. + +.. bpo: 42854 +.. date: 2021-07-20-21-51-35 +.. nonce: ThuDMI +.. section: Library + +Fixed a bug in the :mod:`_ssl` module that was throwing :exc:`OverflowError` +when using :meth:`_ssl._SSLSocket.write` and :meth:`_ssl._SSLSocket.read` +for a big value of the ``len`` parameter. Patch by Pablo Galindo + +.. + +.. bpo: 44686 +.. date: 2021-07-20-19-35-49 +.. nonce: ucCGhu +.. section: Library + +Replace ``unittest.mock._importer`` with ``pkgutil.resolve_name``. + +.. + +.. bpo: 44353 +.. date: 2021-07-20-18-34-16 +.. nonce: ATuYq4 +.. section: Library + +Make ``NewType.__call__`` faster by implementing it in C. Patch provided by +Yurii Karabas. + +.. + +.. bpo: 44682 +.. date: 2021-07-20-00-11-47 +.. nonce: 3m2qVV +.. section: Library + +Change the :mod:`pdb` *commands* directive to disallow setting commands for +an invalid breakpoint and to display an appropriate error. + +.. + +.. bpo: 44353 +.. date: 2021-07-19-22-43-15 +.. nonce: HF81_Q +.. section: Library + +Refactor ``typing.NewType`` from function into callable class. Patch +provided by Yurii Karabas. + +.. + +.. bpo: 44678 +.. date: 2021-07-19-18-45-00 +.. nonce: YMEAu0 +.. section: Library + +Added a separate error message for discontinuous padding in +*binascii.a2b_base64* strict mode. + +.. + +.. bpo: 44524 +.. date: 2021-07-19-14-04-42 +.. nonce: Nbf2JC +.. section: Library + +Add missing ``__name__`` and ``__qualname__`` attributes to ``typing`` +module classes. Patch provided by Yurii Karabas. + +.. + +.. bpo: 40897 +.. date: 2021-07-16-13-40-31 +.. nonce: aveAre +.. section: Library + +Give priority to using the current class constructor in +:func:`inspect.signature`. Patch by Weipeng Hong. + +.. + +.. bpo: 44638 +.. date: 2021-07-16-08-57-27 +.. nonce: EwYKne +.. section: Library + +Add a reference to the zipp project and hint as to how to use it. + +.. + +.. bpo: 44648 +.. date: 2021-07-15-16-51-32 +.. nonce: 2o49TB +.. section: Library + +Fixed wrong error being thrown by :func:`inspect.getsource` when examining a +class in the interactive session. Instead of :exc:`TypeError`, it should be +:exc:`OSError` with appropriate error message. + +.. + +.. bpo: 44608 +.. date: 2021-07-13-09-01-33 +.. nonce: R3IcM1 +.. section: Library + +Fix memory leak in :func:`_tkinter._flatten` if it is called with a sequence +or set, but not list or tuple. + +.. + +.. bpo: 44594 +.. date: 2021-07-12-10-32-48 +.. nonce: eEa5zi +.. section: Library + +Fix an edge case of :class:`ExitStack` and :class:`AsyncExitStack` exception +chaining. They will now match ``with`` block behavior when ``__context__`` +is explicitly set to ``None`` when the exception is in flight. + +.. + +.. bpo: 42799 +.. date: 2021-07-10-19-55-13 +.. nonce: ad4tq8 +.. section: Library + +In :mod:`fnmatch`, the cache size for compiled regex patterns +(:func:`functools.lru_cache`) was bumped up from 256 to 32768, affecting +functions: :func:`fnmatch.fnmatch`, :func:`fnmatch.fnmatchcase`, +:func:`fnmatch.filter`. + +.. + +.. bpo: 41928 +.. date: 2021-07-09-07-14-37 +.. nonce: Q1jMrr +.. section: Library + +Update :func:`shutil.copyfile` to raise :exc:`FileNotFoundError` instead of +confusing :exc:`IsADirectoryError` when a path ending with a +:const:`os.path.sep` does not exist; :func:`shutil.copy` and +:func:`shutil.copy2` are also affected. + +.. + +.. bpo: 44569 +.. date: 2021-07-08-12-22-54 +.. nonce: KZ02v9 +.. section: Library + +Added the :func:`StackSummary.format_frame` function in :mod:`traceback`. +This allows users to customize the way individual lines are formatted in +tracebacks without re-implementing logic to handle recursive tracebacks. + +.. + +.. bpo: 44566 +.. date: 2021-07-05-18-13-25 +.. nonce: o51Bd1 +.. section: Library + +handle StopIteration subclass raised from @contextlib.contextmanager +generator + +.. + +.. bpo: 44558 +.. date: 2021-07-04-21-16-53 +.. nonce: cm7Slv +.. section: Library + +Make the implementation consistency of :func:`~operator.indexOf` between C +and Python versions. Patch by Dong-hee Na. + +.. + +.. bpo: 41249 +.. date: 2021-07-04-11-33-34 +.. nonce: sHdwBE +.. section: Library + +Fixes ``TypedDict`` to work with ``typing.get_type_hints()`` and postponed +evaluation of annotations across modules. + +.. + +.. bpo: 44554 +.. date: 2021-07-02-18-17-56 +.. nonce: aBUmJo +.. section: Library + +Refactor argument processing in :func:`pdb.main` to simplify detection of +errors in input loading and clarify behavior around module or script +invocation. + +.. + +.. bpo: 34798 +.. date: 2021-06-30-13-29-49 +.. nonce: t7FCa0 +.. section: Library + +Break up paragraph about :class:`pprint.PrettyPrinter` construction +parameters to make it easier to read. + +.. + +.. bpo: 44539 +.. date: 2021-06-30-11-34-35 +.. nonce: nP0Xi4 +.. section: Library + +Added support for recognizing JPEG files without JFIF or Exif markers. + +.. + +.. bpo: 44461 +.. date: 2021-06-29-21-17-17 +.. nonce: acqRnV +.. section: Library + +Fix bug with :mod:`pdb`'s handling of import error due to a package which +does not have a ``__main__`` module + +.. + +.. bpo: 43625 +.. date: 2021-06-29-07-27-08 +.. nonce: ZlAxhp +.. section: Library + +Fix a bug in the detection of CSV file headers by +:meth:`csv.Sniffer.has_header` and improve documentation of same. + +.. + +.. bpo: 44516 +.. date: 2021-06-26-12-27-14 +.. nonce: BVyX_y +.. section: Library + +Update vendored pip to 21.1.3 + +.. + +.. bpo: 42892 +.. date: 2021-06-24-19-16-20 +.. nonce: qvRNhI +.. section: Library + +Fixed an exception thrown while parsing a malformed multipart email by +:class:`email.message.EmailMessage`. + +.. + +.. bpo: 44468 +.. date: 2021-06-23-19-02-00 +.. nonce: -klV5- +.. section: Library + +:func:`typing.get_type_hints` now finds annotations in classes and base +classes with unexpected ``__module__``. Previously, it skipped those MRO +elements. + +.. + +.. bpo: 44491 +.. date: 2021-06-23-01-33-01 +.. nonce: tiOlr5 +.. section: Library + +Allow clearing the :mod:`sqlite3` authorizer callback by passing +:const:`None` to :meth:`~sqlite3.Connection.set_authorizer`. Patch by Erlend +E. Aasland. + +.. + +.. bpo: 43977 +.. date: 2021-06-22-16-45-48 +.. nonce: bamAGF +.. section: Library + +Set the proper :const:`Py_TPFLAGS_MAPPING` and :const:`Py_TPFLAGS_SEQUENCE` +flags for subclasses created before a parent has been registered as a +:class:`collections.abc.Mapping` or :class:`collections.abc.Sequence`. + +.. + +.. bpo: 44482 +.. date: 2021-06-22-08-43-04 +.. nonce: U9GznK +.. section: Library + +Fix very unlikely resource leak in :mod:`glob` in alternate Python +implementations. + +.. + +.. bpo: 44466 +.. date: 2021-06-21-12-43-04 +.. nonce: NSm6mv +.. section: Library + +The :mod:`faulthandler` module now detects if a fatal error occurs during a +garbage collector collection. Patch by Victor Stinner. + +.. + +.. bpo: 44471 +.. date: 2021-06-21-10-46-58 +.. nonce: 2QjXv_ +.. section: Library + +A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in +:meth:`contextlib.ExitStack.enter_context` and +:meth:`contextlib.AsyncExitStack.enter_async_context` for objects which do +not support the :term:`context manager` or :term:`asynchronous context +manager` protocols correspondingly. + +.. + +.. bpo: 44404 +.. date: 2021-06-20-19-01-11 +.. nonce: McfrYB +.. section: Library + +:mod:`tkinter`'s ``after()`` method now supports callables without the +``__name__`` attribute. + +.. + +.. bpo: 41546 +.. date: 2021-06-20-14-03-18 +.. nonce: lO1jXU +.. section: Library + +Make :mod:`pprint` (like the builtin ``print``) not attempt to write to +``stdout`` when it is ``None``. + +.. + +.. bpo: 44458 +.. date: 2021-06-20-07-14-46 +.. nonce: myqCQ0 +.. section: Library + +``BUFFER_BLOCK_SIZE`` is now declared static, to avoid linking collisions +when bz2, lmza or zlib are statically linked. + +.. + +.. bpo: 44464 +.. date: 2021-06-19-21-52-27 +.. nonce: U2oa-a +.. section: Library + +Remove exception for flake8 in deprecated importlib.metadata interfaces. +Sync with importlib_metadata 4.6. + +.. + +.. bpo: 44446 +.. date: 2021-06-17-22-39-34 +.. nonce: qwdRic +.. section: Library + +Take into account that ``lineno`` might be ``None`` in +:class:`traceback.FrameSummary`. + +.. + +.. bpo: 44439 +.. date: 2021-06-17-15-01-51 +.. nonce: 1S7QhT +.. section: Library + +Fix in :meth:`bz2.BZ2File.write` / :meth:`lzma.LZMAFile.write` methods, when +the input data is an object that supports the buffer protocol, the file +length may be wrong. + +.. + +.. bpo: 44434 +.. date: 2021-06-16-16-52-14 +.. nonce: SQS4Pg +.. section: Library + +_thread.start_new_thread() no longer calls PyThread_exit_thread() explicitly +at the thread exit, the call was redundant. On Linux with the glibc, +pthread_exit() aborts the whole process if dlopen() fails to open +libgcc_s.so file (ex: EMFILE error). Patch by Victor Stinner. + +.. + +.. bpo: 42972 +.. date: 2021-06-15-13-51-25 +.. nonce: UnyYo1 +.. section: Library + +The _thread.RLock type now fully implement the GC protocol: add a traverse +function and the :const:`Py_TPFLAGS_HAVE_GC` flag. Patch by Victor Stinner. + +.. + +.. bpo: 44422 +.. date: 2021-06-14-23-28-17 +.. nonce: BlWOgv +.. section: Library + +The :func:`threading.enumerate` function now uses a reentrant lock to +prevent a hang on reentrant call. Patch by Victor Stinner. + +.. + +.. bpo: 38291 +.. date: 2021-06-14-14-19-11 +.. nonce: ee4cSX +.. section: Library + +Importing typing.io or typing.re now prints a ``DeprecationWarning``. + +.. + +.. bpo: 37880 +.. date: 2021-06-13-00-16-56 +.. nonce: 5bTrkw +.. section: Library + +argparse actions store_const and append_const each receive a default value +of None when the ``const`` kwarg is not provided. Previously, this raised a +:exc:`TypeError`. + +.. + +.. bpo: 44389 +.. date: 2021-06-12-22-58-20 +.. nonce: WTRnoC +.. section: Library + +Fix deprecation of :data:`ssl.OP_NO_TLSv1_3` + +.. + +.. bpo: 27827 +.. date: 2021-06-12-21-25-35 +.. nonce: TMWh1i +.. section: Library + +:meth:`pathlib.PureWindowsPath.is_reserved` now identifies a greater range +of reserved filenames, including those with trailing spaces or colons. + +.. + +.. bpo: 44395 +.. date: 2021-06-12-10-08-14 +.. nonce: PcW6Sx +.. section: Library + +Fix :meth:`~email.message.MIMEPart.as_string` to pass unixfrom properly. +Patch by Dong-hee Na. + +.. + +.. bpo: 34266 +.. date: 2021-06-10-21-53-46 +.. nonce: k3fxnm +.. section: Library + +Handle exceptions from parsing the arg of :mod:`pdb`'s run/restart command. + +.. + +.. bpo: 44362 +.. date: 2021-06-10-20-07-32 +.. nonce: oVOMfd +.. section: Library + +Improve :mod:`ssl` module's deprecation messages, error reporting, and +documentation for deprecations. + +.. + +.. bpo: 44342 +.. date: 2021-06-10-15-06-47 +.. nonce: qqkGlj +.. section: Library + +[Enum] Change pickling from by-value to by-name. + +.. + +.. bpo: 44356 +.. date: 2021-06-10-08-35-38 +.. nonce: 6oDFhO +.. section: Library + +[Enum] Allow multiple data-type mixins if they are all the same. + +.. + +.. bpo: 44351 +.. date: 2021-06-10-07-26-12 +.. nonce: rvyf2v +.. section: Library + +Restore back :func:`parse_makefile` in :mod:`distutils.sysconfig` because it +behaves differently than the similar implementation in :mod:`sysconfig`. + +.. + +.. bpo: 35800 +.. date: 2021-06-09-10-08-32 +.. nonce: 3hmkWw +.. section: Library + +:class:`smtpd.MailmanProxy` is now removed as it is unusable without an +external module, ``mailman``. Patch by Dong-hee Na. + +.. + +.. bpo: 44357 +.. date: 2021-06-09-08-32-39 +.. nonce: 70Futb +.. section: Library + +Added a function that returns cube root of the given number +:func:`math.cbrt` + +.. + +.. bpo: 44339 +.. date: 2021-06-08-17-47-38 +.. nonce: 9JwMSc +.. section: Library + +Change ``math.pow(?0.0, -math.inf)`` to return ``inf`` instead of raising +``ValueError``. This brings the special-case handling of ``math.pow`` into +compliance with the IEEE 754 standard. + +.. + +.. bpo: 44242 +.. date: 2021-06-07-10-26-14 +.. nonce: MKeMCQ +.. section: Library + +Remove missing flag check from Enum creation and move into a ``verify`` +decorator. + +.. + +.. bpo: 44246 +.. date: 2021-05-31-11-34-56 +.. nonce: yHAkF0 +.. section: Library + +In ``importlib.metadata``, restore compatibility in the result from +``Distribution.entry_points`` (``EntryPoints``) to honor expectations in +older implementations and issuing deprecation warnings for these cases: A. +``EntryPoints`` objects are once again mutable, allowing for ``sort()`` +and other list-based mutation operations. Avoid deprecation warnings by +casting to a mutable sequence (e.g. ``list(dist.entry_points).sort()``). +B. ``EntryPoints`` results once again allow for access by index. To avoid +deprecation warnings, cast the result to a Sequence first (e.g. +``tuple(dist.entry_points)[0]``). + +.. + +.. bpo: 44246 +.. date: 2021-05-31-11-28-03 +.. nonce: nhmt-v +.. section: Library + +In importlib.metadata.entry_points, de-duplication of distributions no +longer requires loading the full metadata for PathDistribution objects, +improving entry point loading performance by ~10x. + +.. + +.. bpo: 43858 +.. date: 2021-05-31-04-51-02 +.. nonce: r7LOu6 +.. section: Library + +Added a function that returns a copy of a dict of logging levels: +:func:`logging.getLevelNamesMapping` + +.. + +.. bpo: 44260 +.. date: 2021-05-30-13-32-09 +.. nonce: ROEbVd +.. section: Library + +The :class:`random.Random` constructor no longer reads system entropy +without need. + +.. + +.. bpo: 44254 +.. date: 2021-05-29-01-05-43 +.. nonce: f06xDm +.. section: Library + +On Mac, give turtledemo button text a color that works on both light or dark +background. Programmers cannot control the latter. + +.. + +.. bpo: 44258 +.. date: 2021-05-28-09-43-33 +.. nonce: nh5F7R +.. section: Library + +Support PEP 515 for Fraction's initialization from string. + +.. + +.. bpo: 44235 +.. date: 2021-05-26-22-04-40 +.. nonce: qFBYpp +.. section: Library + +Remove deprecated functions in the :mod:`gettext`. Patch by Dong-hee Na. + +.. + +.. bpo: 38693 +.. date: 2021-05-26-14-50-06 +.. nonce: NkMacJ +.. section: Library + +Prefer f-strings to ``.format`` in importlib.resources. + +.. + +.. bpo: 33693 +.. date: 2021-05-26-13-34-37 +.. nonce: 3okzdo +.. section: Library + +Importlib.metadata now prefers f-strings to .format. + +.. + +.. bpo: 44241 +.. date: 2021-05-26-13-15-51 +.. nonce: TBqej8 +.. section: Library + +Incorporate minor tweaks from importlib_metadata 4.1: SimplePath protocol, +support for Metadata 2.2. + +.. + +.. bpo: 43216 +.. date: 2021-05-25-23-26-38 +.. nonce: xTUyyX +.. section: Library + +Remove the :func:`@asyncio.coroutine ` :term:`decorator` +enabling legacy generator-based coroutines to be compatible with async/await +code; remove :class:`asyncio.coroutines.CoroWrapper` used for wrapping +legacy coroutine objects in the debug mode. The decorator has been +deprecated since Python 3.8 and the removal was initially scheduled for +Python 3.10. Patch by Illia Volochii. + +.. + +.. bpo: 44210 +.. date: 2021-05-21-21-23-43 +.. nonce: 5afQ3K +.. section: Library + +Make importlib.metadata._meta.PackageMetadata public. + +.. + +.. bpo: 43643 +.. date: 2021-05-21-12-12-35 +.. nonce: GWnmcF +.. section: Library + +Declare readers.MultiplexedPath.name as a property per the spec. + +.. + +.. bpo: 27334 +.. date: 2021-05-18-00-17-21 +.. nonce: 32EJZi +.. section: Library + +The :mod:`sqlite3` context manager now performs a rollback (thus releasing +the database lock) if commit failed. Patch by Luca Citi and Erlend E. +Aasland. + +.. + +.. bpo: 4928 +.. date: 2021-05-17-21-05-06 +.. nonce: Ot2yjO +.. section: Library + +Documented existing behavior on POSIX: NamedTemporaryFiles are not deleted +when creating process is killed with SIGKILL + +.. + +.. bpo: 44154 +.. date: 2021-05-17-07-24-24 +.. nonce: GRI5bf +.. section: Library + +Optimize :class:`fractions.Fraction` pickling for large components. + +.. + +.. bpo: 33433 +.. date: 2021-05-16-17-48-24 +.. nonce: MyzO71 +.. section: Library + +For IPv4 mapped IPv6 addresses (:rfc:`4291` Section 2.5.5.2), the +:mod:`ipaddress.IPv6Address.is_private` check is deferred to the mapped IPv4 +address. This solves a bug where public mapped IPv4 addresses were +considered private by the IPv6 check. + +.. + +.. bpo: 44150 +.. date: 2021-05-16-11-57-38 +.. nonce: xAhhik +.. section: Library + +Add optional *weights* argument to statistics.fmean(). + +.. + +.. bpo: 44142 +.. date: 2021-05-16-02-24-23 +.. nonce: t-XU8k +.. section: Library + +:func:`ast.unparse` will now drop the redundant parentheses when tuples used +as assignment targets (e.g in for loops). + +.. + +.. bpo: 44145 +.. date: 2021-05-16-00-00-38 +.. nonce: ko5SJ7 +.. section: Library + +:mod:`hmac` computations were not releasing the GIL while calling the +OpenSSL ``HMAC_Update`` C API (a new feature in 3.9). This unintentionally +prevented parallel computation as other :mod:`hashlib` algorithms support. + +.. + +.. bpo: 44095 +.. date: 2021-05-14-16-06-02 +.. nonce: v_pLwY +.. section: Library + +:class:`zipfile.Path` now supports :attr:`zipfile.Path.stem`, +:attr:`zipfile.Path.suffixes`, and :attr:`zipfile.Path.suffix` attributes. + +.. + +.. bpo: 44077 +.. date: 2021-05-13-19-44-38 +.. nonce: 04b2a4 +.. section: Library + +It's now possible to receive the type of service (ToS), a.k.a. +differentiated services (DS), a.k.a. differenciated services code point +(DSCP) and excplicit congestion notification (ECN) IP header fields with +``socket.IP_RECVTOS``. + +.. + +.. bpo: 37788 +.. date: 2021-05-13-19-07-28 +.. nonce: adeFcf +.. section: Library + +Fix a reference leak when a Thread object is never joined. + +.. + +.. bpo: 38908 +.. date: 2021-05-12-16-43-21 +.. nonce: nM2_rO +.. section: Library + +Subclasses of ``typing.Protocol`` which only have data variables declared +will now raise a ``TypeError`` when checked with ``isinstance`` unless they +are decorated with :func:`runtime_checkable`. Previously, these checks +passed silently. Patch provided by Yurii Karabas. + +.. + +.. bpo: 44098 +.. date: 2021-05-10-17-45-00 +.. nonce: _MoxuZ +.. section: Library + +``typing.ParamSpec`` will no longer be found in the ``__parameters__`` of +most :mod:`typing` generics except in valid use locations specified by +:pep:`612`. This prevents incorrect usage like ``typing.List[P][int]``. This +change means incorrect usage which may have passed silently in 3.10 beta 1 +and earlier will now error. + +.. + +.. bpo: 44089 +.. date: 2021-05-09-22-52-34 +.. nonce: IoANsN +.. section: Library + +Allow subclassing ``csv.Error`` in 3.10 (it was allowed in 3.9 and earlier +but was disallowed in early versions of 3.10). + +.. + +.. bpo: 44081 +.. date: 2021-05-09-03-26-31 +.. nonce: A-Mrto +.. section: Library + +:func:`ast.unparse` now doesn't use redundant spaces to separate ``lambda`` +and the ``:`` if there are no parameters. + +.. + +.. bpo: 44061 +.. date: 2021-05-07-08-39-23 +.. nonce: MvElG6 +.. section: Library + +Fix regression in previous release when calling :func:`pkgutil.iter_modules` +with a list of :class:`pathlib.Path` objects + +.. + +.. bpo: 44059 +.. date: 2021-05-06-16-01-55 +.. nonce: GF5r6O +.. section: Library + +Register the SerenityOS Browser in the :mod:`webbrowser` module. + +.. + +.. bpo: 36515 +.. date: 2021-05-05-11-44-49 +.. nonce: uOSa3q +.. section: Library + +The :mod:`hashlib` module no longer does unaligned memory accesses when +compiled for ARM platforms. + +.. + +.. bpo: 40465 +.. date: 2021-05-03-19-59-14 +.. nonce: 1tB4Y0 +.. section: Library + +Remove random module features deprecated in Python 3.9. + +.. + +.. bpo: 44018 +.. date: 2021-05-03-10-07-43 +.. nonce: VDyW8f +.. section: Library + +random.seed() no longer mutates bytearray inputs. + +.. + +.. bpo: 38352 +.. date: 2021-05-02-13-54-25 +.. nonce: N9MlhV +.. section: Library + +Add ``IO``, ``BinaryIO``, ``TextIO``, ``Match``, and ``Pattern`` to +``typing.__all__``. Patch by Jelle Zijlstra. + +.. + +.. bpo: 44002 +.. date: 2021-05-01-15-43-37 +.. nonce: KLT_wd +.. section: Library + +:mod:`urllib.parse` now uses :func:`functool.lru_cache` for its internal URL +splitting and quoting caches instead of rolling its own like its the '90s. + +The undocumented internal :mod:`urllib.parse` ``Quoted`` class API is now +deprecated, for removal in 3.14. + +.. + +.. bpo: 43972 +.. date: 2021-04-30-16-58-24 +.. nonce: Y2r9lg +.. section: Library + +When :class:`http.server.SimpleHTTPRequestHandler` sends a ``301 (Moved +Permanently)`` for a directory path not ending with `/`, add a +``Content-Length: 0`` header. This improves the behavior for certain +clients. + +.. + +.. bpo: 28528 +.. date: 2021-04-29-00-48-00 +.. nonce: JLAVWj +.. section: Library + +Fix a bug in :mod:`pdb` where :meth:`~pdb.Pdb.checkline` raises +:exc:`AttributeError` if it is called after :meth:`~pdb.Pdb.reset`. + +.. + +.. bpo: 43853 +.. date: 2021-04-15-12-02-17 +.. nonce: XXCVAp +.. section: Library + +Improved string handling for :mod:`sqlite3` user-defined functions and +aggregates: + +* It is now possible to pass strings with embedded null characters to UDFs +* Conversion failures now correctly raise :exc:`MemoryError` + +Patch by Erlend E. Aasland. + +.. + +.. bpo: 43666 +.. date: 2021-03-30-08-39-08 +.. nonce: m72tlH +.. section: Library + +AIX: `Lib/_aix_support.get_platform()` may fail in an AIX WPAR. The fileset +bos.rte appears to have a builddate in both LPAR and WPAR so this fileset is +queried rather than bos.mp64. To prevent a similiar situation (no builddate +in ODM) a value (9988) sufficient for completing a build is provided. Patch +by M Felt. + +.. + +.. bpo: 43650 +.. date: 2021-03-29-00-23-30 +.. nonce: v01tic +.. section: Library + +Fix :exc:`MemoryError` in :func:`shutil.unpack_archive` which fails inside +:func:`shutil._unpack_zipfile` on large files. Patch by Igor Bolshakov. + +.. + +.. bpo: 43612 +.. date: 2021-03-24-09-40-02 +.. nonce: vMGZ4y +.. section: Library + +:func:`zlib.compress` now accepts a wbits parameter which allows users to +compress data as a raw deflate block without zlib headers and trailers in +one go. Previously this required instantiating a ``zlib.compressobj``. It +also provides a faster alternative to ``gzip.compress`` when wbits=31 is +used. + +.. + +.. bpo: 43392 +.. date: 2021-03-03-13-32-37 +.. nonce: QQumou +.. section: Library + +:func:`importlib._bootstrap._find_and_load` now implements a two-step check +to avoid locking when modules have been already imported and are ready. This +improves performance of repeated calls to :func:`importlib.import_module` +and :func:`importlib.__import__`. + +.. + +.. bpo: 43318 +.. date: 2021-02-25-08-32-06 +.. nonce: bZJw6V +.. section: Library + +Fix a bug where :mod:`pdb` does not always echo cleared breakpoints. + +.. + +.. bpo: 43234 +.. date: 2021-02-15-22-14-31 +.. nonce: F-vKAT +.. section: Library + +Prohibit passing non-:class:`concurrent.futures.ThreadPoolExecutor` +executors to :meth:`loop.set_default_executor` following a deprecation in +Python 3.8. Patch by Illia Volochii. + +.. + +.. bpo: 43232 +.. date: 2021-02-15-21-17-46 +.. nonce: awc4yZ +.. section: Library + +Prohibit previously deprecated potentially disruptive operations on +:class:`asyncio.trsock.TransportSocket`. Patch by Illia Volochii. + +.. + +.. bpo: 30077 +.. date: 2021-02-04-23-16-03 +.. nonce: v6TqAi +.. section: Library + +Added support for Apple's aifc/sowt pseudo-compression + +.. + +.. bpo: 42971 +.. date: 2021-02-02-20-11-14 +.. nonce: OpVoFu +.. section: Library + +Add definition of ``errno.EQFULL`` for platforms that define this constant +(such as macOS). + +.. + +.. bpo: 43086 +.. date: 2021-01-31-18-24-54 +.. nonce: 2_P-SH +.. section: Library + +Added a new optional :code:`strict_mode` parameter to *binascii.a2b_base64*. +When :code:`scrict_mode` is set to :code:`True`, the *a2b_base64* function +will accept only valid base64 content. More details about what "valid base64 +content" is, can be found in the function's documentation. + +.. + +.. bpo: 43024 +.. date: 2021-01-25-21-24-55 +.. nonce: vAUrIi +.. section: Library + +Improve the help signature of :func:`traceback.print_exception`, +:func:`traceback.format_exception` and +:func:`traceback.format_exception_only`. + +.. + +.. bpo: 33809 +.. date: 2021-01-16-18-36-00 +.. nonce: BiMK6V +.. section: Library + +Add the :meth:`traceback.TracebackException.print` method which prints the +formatted exception information. + +.. + +.. bpo: 42862 +.. date: 2021-01-13-00-02-44 +.. nonce: Z6ACLN +.. section: Library + +:mod:`sqlite3` now utilizes :meth:`functools.lru_cache` to implement the +connection statement cache. As a small optimisation, the default statement +cache size has been increased from 100 to 128. Patch by Erlend E. Aasland. + +.. + +.. bpo: 41818 +.. date: 2020-12-08-01-08-58 +.. nonce: zO8vV7 +.. section: Library + +Soumendra Ganguly: add termios.tcgetwinsize(), termios.tcsetwinsize(). + +.. + +.. bpo: 40497 +.. date: 2020-10-18-09-42-53 +.. nonce: CRz2sG +.. section: Library + +:meth:`subprocess.check_output` now raises :exc:`ValueError` when the +invalid keyword argument *check* is passed by user code. Previously such use +would fail later with a :exc:`TypeError`. Patch by R?mi Lapeyre. + +.. + +.. bpo: 37449 +.. date: 2020-10-11-20-23-48 +.. nonce: f-t3V6 +.. section: Library + +``ensurepip`` now uses ``importlib.resources.files()`` traversable APIs + +.. + +.. bpo: 40956 +.. date: 2020-10-01-21-46-34 +.. nonce: _tvsZ7 +.. section: Library + +Use Argument Clinic in :mod:`sqlite3`. Patches by Erlend E. Aasland. + +.. + +.. bpo: 41730 +.. date: 2020-09-10-07-23-24 +.. nonce: DyKFi9 +.. section: Library + +``DeprecationWarning`` is now raised when importing :mod:`tkinter.tix`, +which has been deprecated in documentation since Python 3.6. + +.. + +.. bpo: 20684 +.. date: 2020-07-30-14-37-15 +.. nonce: qV35GU +.. section: Library + +Remove unused ``_signature_get_bound_param`` function from :mod:`inspect` - +by Anthony Sottile. + +.. + +.. bpo: 41402 +.. date: 2020-07-26-18-17-30 +.. nonce: YRkVkp +.. section: Library + +Fix :meth:`email.message.EmailMessage.set_content` when called with binary +data and ``7bit`` content transfer encoding. + +.. + +.. bpo: 32695 +.. date: 2020-07-13-23-46-59 +.. nonce: tTqqXe +.. section: Library + +The *compresslevel* and *preset* keyword arguments of :func:`tarfile.open` +are now both documented and tested. + +.. + +.. bpo: 41137 +.. date: 2020-07-01-17-42-41 +.. nonce: AnqbP- +.. section: Library + +Use utf-8 encoding while reading .pdbrc files. Patch by Srinivas Reddy +Thatiparthy + +.. + +.. bpo: 24391 +.. date: 2020-05-30-10-48-04 +.. nonce: ZCTnhX +.. section: Library + +Improved reprs of :mod:`threading` synchronization objects: +:class:`~threading.Semaphore`, :class:`~threading.BoundedSemaphore`, +:class:`~threading.Event` and :class:`~threading.Barrier`. + +.. + +.. bpo: 5846 +.. date: 2020-05-25-23-58-29 +.. nonce: O9BIfm +.. section: Library + +Deprecated the following :mod:`unittest` functions, scheduled for removal in +Python 3.13: + +* :func:`~unittest.findTestCases` +* :func:`~unittest.makeSuite` +* :func:`~unittest.getTestCaseNames` + +Use :class:`~unittest.TestLoader` methods instead: + +* :meth:`unittest.TestLoader.loadTestsFromModule` +* :meth:`unittest.TestLoader.loadTestsFromTestCase` +* :meth:`unittest.TestLoader.getTestCaseNames` + +Patch by Erlend E. Aasland. + +.. + +.. bpo: 40563 +.. date: 2020-05-21-01-42-32 +.. nonce: fDn5bP +.. section: Library + +Support pathlike objects on dbm/shelve. Patch by Hakan ?elik and +Henry-Joseph Aud?oud. + +.. + +.. bpo: 34990 +.. date: 2020-04-24-20-39-38 +.. nonce: 3SmL9M +.. section: Library + +Fixed a Y2k38 bug in the compileall module where it would fail to compile +files with a modification time after the year 2038. + +.. + +.. bpo: 39549 +.. date: 2020-02-03-21-18-31 +.. nonce: l4a8uH +.. section: Library + +Whereas the code for reprlib.Repr had previously used a hardcoded string +value of '...', this PR updates it to use of a ?fillvalue? attribute, whose +value defaults to '...' and can be reset in either individual reprlib.Repr +instances or in subclasses thereof. + +.. + +.. bpo: 37022 +.. date: 2020-01-25-12-58-20 +.. nonce: FUZI25 +.. section: Library + +:mod:`pdb` now displays exceptions from ``repr()`` with its ``p`` and ``pp`` +commands. + +.. + +.. bpo: 38840 +.. date: 2020-01-16-23-41-16 +.. nonce: VzzYZz +.. section: Library + +Fix ``test___all__`` on platforms lacking a shared memory implementation. + +.. + +.. bpo: 39359 +.. date: 2020-01-16-13-54-28 +.. nonce: hzTu0h +.. section: Library + +Add one missing check that the password is a bytes object for an encrypted +zipfile. + +.. + +.. bpo: 38741 +.. date: 2019-11-12-18-59-33 +.. nonce: W7IYkq +.. section: Library + +:mod:`configparser`: using ']' inside a section header will no longer cut +the section name short at the ']' + +.. + +.. bpo: 38415 +.. date: 2019-10-08-14-08-59 +.. nonce: N1bUw6 +.. section: Library + +Added missing behavior to :func:`contextlib.asynccontextmanager` to match +:func:`contextlib.contextmanager` so decorated functions can themselves be +decorators. + +.. + +.. bpo: 30256 +.. date: 2019-09-25-13-54-41 +.. nonce: wBkzox +.. section: Library + +Pass multiprocessing BaseProxy argument ``manager_owned`` through AutoProxy. + +.. + +.. bpo: 27513 +.. date: 2019-06-03-23-53-25 +.. nonce: qITN7d +.. section: Library + +:func:`email.utils.getaddresses` now accepts :class:`email.header.Header` +objects along with string values. Patch by Zackery Spytz. + +.. + +.. bpo: 16379 +.. date: 2019-05-08-15-14-32 +.. nonce: rN5JVe +.. section: Library + +Add SQLite error code and name to :mod:`sqlite3` exceptions. Patch by Aviv +Palivoda, Daniel Shahaf, and Erlend E. Aasland. + +.. + +.. bpo: 26228 +.. date: 2019-02-26-09-31-59 +.. nonce: wyrHKc +.. section: Library + +pty.spawn no longer hangs on FreeBSD, macOS, and Solaris. + +.. + +.. bpo: 33349 +.. date: 2018-04-24-14-25-07 +.. nonce: Y_0LIr +.. section: Library + +lib2to3 now recognizes async generators everywhere. + +.. + +.. bpo: 29298 +.. date: 2017-09-20-14-43-03 +.. nonce: _78CSN +.. section: Library + +Fix ``TypeError`` when required subparsers without ``dest`` do not receive +arguments. Patch by Anthony Sottile. + +.. + +.. bpo: 45216 +.. date: 2021-09-18-13-45-19 +.. nonce: o56nyt +.. section: Documentation + +Remove extra documentation listing methods in ``difflib``. It was rendering +twice in pydoc and was outdated in some places. + +.. + +.. bpo: 45024 +.. date: 2021-09-08-17-20-19 +.. nonce: dkNPNi +.. section: Documentation + +:mod:`collections.abc` documentation has been expanded to explicitly cover +how instance and subclass checks work, with additional doctest examples and +an exhaustive list of ABCs which test membership purely by presence of the +right :term:`special method`\s. Patch by Raymond Hettinger. + +.. + +.. bpo: 44957 +.. date: 2021-08-19-15-53-08 +.. nonce: imqrh3 +.. section: Documentation + +Promote PEP 604 union syntax by using it where possible. Also, mention ``X | +Y`` more prominently in section about ``Union`` and mention ``X | None`` at +all in section about ``Optional``. + +.. + +.. bpo: 16580 +.. date: 2021-08-13-20-17-59 +.. nonce: MZ_iK9 +.. section: Documentation + +Added code equivalents for the :meth:`int.to_bytes` and +:meth:`int.from_bytes` methods, as well as tests ensuring that these code +equivalents are valid. + +.. + +.. bpo: 44903 +.. date: 2021-08-13-19-08-03 +.. nonce: aJuvQF +.. section: Documentation + +Removed the othergui.rst file, any references to it, and the list of GUI +frameworks in the FAQ. In their place I've added links to the Python Wiki +`page on GUI frameworks `. + +.. + +.. bpo: 33479 +.. date: 2021-08-11-18-02-06 +.. nonce: rCe4c5 +.. section: Documentation + +Tkinter documentation has been greatly expanded with new "Architecture" and +"Threading model" sections. + +.. + +.. bpo: 36700 +.. date: 2021-08-09-19-58-45 +.. nonce: WPNW5f +.. section: Documentation + +:mod:`base64` RFC references were updated to point to :rfc:`4648`; a section +was added to point users to the new "security considerations" section of the +RFC. + +.. + +.. bpo: 44740 +.. date: 2021-07-26-23-48-31 +.. nonce: zMFGMV +.. section: Documentation + +Replaced occurences of uppercase "Web" and "Internet" with lowercase +versions per the 2016 revised Associated Press Style Book. + +.. + +.. bpo: 44693 +.. date: 2021-07-25-23-04-15 +.. nonce: JuCbNq +.. section: Documentation + +Update the definition of __future__ in the glossary by replacing the +confusing word "pseudo-module" with a more accurate description. + +.. + +.. bpo: 35183 +.. date: 2021-07-22-08-28-03 +.. nonce: p9BWTB +.. section: Documentation + +Add typical examples to os.path.splitext docs + +.. + +.. bpo: 30511 +.. date: 2021-07-20-21-03-18 +.. nonce: eMFkRi +.. section: Documentation + +Clarify that :func:`shutil.make_archive` is not thread-safe due to reliance +on changing the current working directory. + +.. + +.. bpo: 44561 +.. date: 2021-07-18-22-43-14 +.. nonce: T7HpWm +.. section: Documentation + +Update of three expired hyperlinks in Doc/distributing/index.rst: "Project +structure", "Building and packaging the project", and "Uploading the project +to the Python Packaging Index". + +.. + +.. bpo: 44651 +.. date: 2021-07-18-22-26-02 +.. nonce: SjT9iY +.. section: Documentation + +Delete entry "coercion" in Doc/glossary.rst for its outdated definition. + +.. + +.. bpo: 42958 +.. date: 2021-07-15-11-19-03 +.. nonce: gC5IHM +.. section: Documentation + +Updated the docstring and docs of :func:`filecmp.cmp` to be more accurate +and less confusing especially in respect to *shallow* arg. + +.. + +.. bpo: 44631 +.. date: 2021-07-13-22-25-13 +.. nonce: qkGwe4 +.. section: Documentation + +Refactored the ``repr()`` code of the ``_Environ`` (os module). + +.. + +.. bpo: 44613 +.. date: 2021-07-12-11-39-20 +.. nonce: DIXNzc +.. section: Documentation + +importlib.metadata is no longer provisional. + +.. + +.. bpo: 44558 +.. date: 2021-07-03-18-25-17 +.. nonce: 0pTknl +.. section: Documentation + +Match the docstring and python implementation of :func:`~operator.countOf` +to the behavior of its c implementation. + +.. + +.. bpo: 44544 +.. date: 2021-07-02-14-02-29 +.. nonce: _5_aCz +.. section: Documentation + +List all kwargs for :func:`textwrap.wrap`, :func:`textwrap.fill`, and +:func:`textwrap.shorten`. Now, there are nav links to attributes of +:class:`TextWrap`, which makes navigation much easier while minimizing +duplication in the documentation. + +.. + +.. bpo: 38062 +.. date: 2021-06-28-12-13-48 +.. nonce: 9Ehp9O +.. section: Documentation + +Clarify that atexit uses equality comparisons internally. + +.. + +.. bpo: 40620 +.. date: 2021-06-26-17-41-06 +.. nonce: PAYDrB +.. section: Documentation + +Convert examples in tutorial controlflow.rst section 4.3 to be +interpreter-demo style. + +.. + +.. bpo: 43066 +.. date: 2021-06-24-14-37-16 +.. nonce: Ti7ahX +.. section: Documentation + +Added a warning to :mod:`zipfile` docs: filename arg with a leading slash +may cause archive to be un-openable on Windows systems. + +.. + +.. bpo: 39452 +.. date: 2021-06-23-15-21-36 +.. nonce: o_I-6d +.. section: Documentation + +Rewrote ``Doc/library/__main__.rst``. Broadened scope of the document to +explicitly discuss and differentiate between ``__main__.py`` in packages +versus the ``__name__ == '__main__'`` expression (and the idioms that +surround it). + +.. + +.. bpo: 13814 +.. date: 2021-06-21-15-46-32 +.. nonce: LDcslu +.. section: Documentation + +In the Design FAQ, answer "Why don't generators support the with statement?" + +.. + +.. bpo: 27752 +.. date: 2021-06-18-18-04-53 +.. nonce: NEByNk +.. section: Documentation + +Documentation of csv.Dialect is more descriptive. + +.. + +.. bpo: 44453 +.. date: 2021-06-18-06-44-45 +.. nonce: 3PIkj2 +.. section: Documentation + +Fix documentation for the return type of :func:`sysconfig.get_path`. + +.. + +.. bpo: 44392 +.. date: 2021-06-16-18-09-49 +.. nonce: 6RF1Sc +.. section: Documentation + +Added a new section in the C API documentation for types used in type +hinting. Documented ``Py_GenericAlias`` and ``Py_GenericAliasType``. + +.. + +.. bpo: 38291 +.. date: 2021-06-14-09-20-37 +.. nonce: VMYa_Q +.. section: Documentation + +Mark ``typing.io`` and ``typing.re`` as deprecated since Python 3.8 in the +documentation. They were never properly supported by type checkers. + +.. + +.. bpo: 44322 +.. date: 2021-06-06-14-12-00 +.. nonce: K0PHfE +.. section: Documentation + +Document that SyntaxError args have a details tuple and that details are +adjusted for errors in f-string field replacement expressions. + +.. + +.. bpo: 42392 +.. date: 2021-05-26-11-16-33 +.. nonce: oxRx6E +.. section: Documentation + +Document the deprecation and removal of the ``loop`` parameter for many +functions and classes in :mod:`asyncio`. + +.. + +.. bpo: 44195 +.. date: 2021-05-23-09-11-28 +.. nonce: 1bqkOs +.. section: Documentation + +Corrected references to ``TraversableResources`` in docs. There is no +``TraversableReader``. + +.. + +.. bpo: 41963 +.. date: 2021-05-17-20-03-47 +.. nonce: eUz9_o +.. section: Documentation + +Document that ``ConfigParser`` strips off comments when reading +configuration files. + +.. + +.. bpo: 44072 +.. date: 2021-05-08-09-48-05 +.. nonce: fb2x5I +.. section: Documentation + +Correct where in the numeric ABC hierarchy ``**`` support is added, i.e., in +numbers.Complex, not numbers.Integral. + +.. + +.. bpo: 43558 +.. date: 2021-05-07-12-27-09 +.. nonce: UGhA8R +.. section: Documentation + +Add the remark to :mod:`dataclasses` documentation that the :meth:`__init__` +of any base class has to be called in :meth:`__post_init__`, along with a +code example. + +.. + +.. bpo: 44025 +.. date: 2021-05-03-22-08-08 +.. nonce: gcB7iP +.. section: Documentation + +Clarify when '_' in match statements is a keyword, and when not. + +.. + +.. bpo: 41706 +.. date: 2020-09-03-13-37-19 +.. nonce: _zXWOR +.. section: Documentation + +Fix docs about how methods like ``__add__`` are invoked when evaluating +operator expressions. + +.. + +.. bpo: 41621 +.. date: 2020-08-24-13-35-04 +.. nonce: nqaw9G +.. section: Documentation + +Document that :class:`collections.defaultdict` parameter ``default_factory`` +defaults to None and is positional-only. + +.. + +.. bpo: 41576 +.. date: 2020-08-21-22-59-37 +.. nonce: 7a6CQR +.. section: Documentation + +document BaseException in favor of bare except + +.. + +.. bpo: 21760 +.. date: 2020-03-21-01-19-28 +.. nonce: CqofIc +.. section: Documentation + +The description for __file__ fixed. Patch by Furkan Onder + +.. + +.. bpo: 39498 +.. date: 2020-01-30-05-18-48 +.. nonce: Nu3sFL +.. section: Documentation + +Add a "Security Considerations" index which links to standard library +modules that have explicitly documented security considerations. + +.. + +.. bpo: 33479 +.. date: 2018-05-19-15-59-29 +.. nonce: 4cLlxo +.. section: Documentation + +Remove the unqualified claim that tkinter is threadsafe. It has not been +true for several years and likely never was. An explanation of what is true +may be added later, after more discussion, and possibly after patching +_tkinter.c, + +.. + +.. bpo: 40173 +.. date: 2021-09-30-16-54-39 +.. nonce: J_slCw +.. section: Tests + +Fix :func:`test.support.import_helper.import_fresh_module`. + +.. + +.. bpo: 45280 +.. date: 2021-09-25-11-05-31 +.. nonce: 3MA6lC +.. section: Tests + +Add a test case for empty :class:`typing.NamedTuple`. + +.. + +.. bpo: 45269 +.. date: 2021-09-24-10-41-49 +.. nonce: 8jKEr8 +.. section: Tests + +Cover case when invalid ``markers`` type is supplied to ``c_make_encoder``. + +.. + +.. bpo: 45128 +.. date: 2021-09-16-17-22-35 +.. nonce: Jz6fl2 +.. section: Tests + +Fix ``test_multiprocessing_fork`` failure due to ``test_logging`` and +``sys.modules`` manipulation. + +.. + +.. bpo: 45209 +.. date: 2021-09-15-23-32-39 +.. nonce: 55ntL5 +.. section: Tests + +Fix ``UserWarning: resource_tracker`` warning in +``_test_multiprocessing._TestSharedMemory.test_shared_memory_cleaned_after_process_termination`` + +.. + +.. bpo: 45185 +.. date: 2021-09-14-14-54-04 +.. nonce: qFx5I6 +.. section: Tests + +Enables ``TestEnumerations`` test cases in ``test_ssl`` suite. + +.. + +.. bpo: 45195 +.. date: 2021-09-14-13-16-18 +.. nonce: EyQR1G +.. section: Tests + +Fix test_readline.test_nonascii(): sometimes, the newline character is not +written at the end, so don't expect it in the output. Patch by Victor +Stinner. + +.. + +.. bpo: 45156 +.. date: 2021-09-13-00-28-17 +.. nonce: 8oomV3 +.. section: Tests + +Fixes infinite loop on :func:`unittest.mock.seal` of mocks created by +:func:`~unittest.create_autospec`. + +.. + +.. bpo: 45125 +.. date: 2021-09-11-22-08-18 +.. nonce: FVSzs2 +.. section: Tests + +Improves pickling tests and docs of ``SharedMemory`` and ``SharableList`` +objects. + +.. + +.. bpo: 44860 +.. date: 2021-09-08-13-01-37 +.. nonce: qXd0kx +.. section: Tests + +Update ``test_sysconfig.test_user_similar()`` for the posix_user scheme: +``platlib`` doesn't use :data:`sys.platlibdir`. Patch by Victor Stinner. + +.. + +.. bpo: 45052 +.. date: 2021-09-06-19-00-29 +.. nonce: yrOK3J +.. section: Tests + +``WithProcessesTestSharedMemory.test_shared_memory_basics`` test was +ignored, because ``self.assertEqual(sms.size, sms2.size)`` line was failing. +It is now removed and test is unskipped. + +The main motivation for this line to be removed from the test is that the +``size`` of ``SharedMemory`` is not ever guaranteed to be the same. It is +decided by the platform. + +.. + +.. bpo: 44895 +.. date: 2021-09-01-17-17-44 +.. nonce: kV7H77 +.. section: Tests + +libregrtest now clears the type cache later to reduce the risk of false +alarm when checking for reference leaks. Previously, the type cache was +cleared too early and libregrtest raised a false alarm about reference leaks +under very specific conditions. Patch by Irit Katriel and Victor Stinner. + +.. + +.. bpo: 45042 +.. date: 2021-08-30-11-54-14 +.. nonce: QMz3X8 +.. section: Tests + +Fixes that test classes decorated with +``@hashlib_helper.requires_hashdigest`` were skipped all the time. + +.. + +.. bpo: 25130 +.. date: 2021-08-27-22-37-19 +.. nonce: ig4oJe +.. section: Tests + +Add calls of :func:`gc.collect` in tests to support PyPy. + +.. + +.. bpo: 45011 +.. date: 2021-08-26-14-20-44 +.. nonce: mQZdXU +.. section: Tests + +Made tests relying on the :mod:`_asyncio` C extension module optional to +allow running on alternative Python implementations. Patch by Serhiy +Storchaka. + +.. + +.. bpo: 44949 +.. date: 2021-08-18-18-30-12 +.. nonce: VE5ENv +.. section: Tests + +Fix auto history tests of test_readline: sometimes, the newline character is +not written at the end, so don't expect it in the output. + +.. + +.. bpo: 44891 +.. date: 2021-08-13-12-11-06 +.. nonce: T9_mBT +.. section: Tests + +Tests were added to clarify :func:`id` is preserved when ``obj * 1`` is used +on :class:`str` and :class:`bytes` objects. Patch by Nikita Sobolev. + +.. + +.. bpo: 44852 +.. date: 2021-08-06-18-36-04 +.. nonce: sUL8YX +.. section: Tests + +Add ability to wholesale silence DeprecationWarnings while running the +regression test suite. + +.. + +.. bpo: 40928 +.. date: 2021-08-06-00-07-15 +.. nonce: aIwb6G +.. section: Tests + +Notify users running test_decimal regression tests on macOS of potential +harmless "malloc can't allocate region" messages spewed by test_decimal. + +.. + +.. bpo: 44734 +.. date: 2021-07-24-20-09-15 +.. nonce: KKsNOV +.. section: Tests + +Fixed floating point precision issue in turtle tests. + +.. + +.. bpo: 44708 +.. date: 2021-07-22-16-38-39 +.. nonce: SYNaac +.. section: Tests + +Regression tests, when run with -w, are now re-running only the affected +test methods instead of re-running the entire test file. + +.. + +.. bpo: 42095 +.. date: 2021-07-17-11-41-20 +.. nonce: kCB7oj +.. section: Tests + +Added interop tests for Apple plists: generate plist files with Python +plistlib and parse with Apple plutil; and the other way round. + +.. + +.. bpo: 44647 +.. date: 2021-07-16-14-02-33 +.. nonce: 5LzqIy +.. section: Tests + +Added a permanent Unicode-valued environment variable to regression tests to +ensure they handle this use case in the future. If your test environment +breaks because of that, report a bug to us, and temporarily set +PYTHONREGRTEST_UNICODE_GUARD=0 in your test environment. + +.. + +.. bpo: 44515 +.. date: 2021-06-26-18-37-36 +.. nonce: e9fO6f +.. section: Tests + +Adjust recently added contextlib tests to avoid assuming the use of a +refcounted GC + +.. + +.. bpo: 44287 +.. date: 2021-06-21-17-53-41 +.. nonce: YON57s +.. section: Tests + +Fix asyncio test_popen() of test_windows_utils by using a longer timeout. +Use military grade battle-tested :data:`test.support.SHORT_TIMEOUT` timeout +rather than a hardcoded timeout of 10 seconds: it's 30 seconds by default, +but it is made longer on slow buildbots. Patch by Victor Stinner. + +.. + +.. bpo: 44451 +.. date: 2021-06-18-15-19-35 +.. nonce: aj5pqE +.. section: Tests + +Reset ``DeprecationWarning`` filters in +``test.test_importlib.test_metadata_api.APITests.test_entry_points_by_index`` +to avoid ``StopIteration`` error if ``DeprecationWarnings`` are ignored. + +.. + +.. bpo: 44363 +.. date: 2021-06-10-11-19-43 +.. nonce: -K9jD0 +.. section: Tests + +Account for address sanitizer in test_capi. test_capi now passes when run +GCC address sanitizer. + +.. + +.. bpo: 44364 +.. date: 2021-06-09-15-32-05 +.. nonce: zu9Zee +.. section: Tests + +Add non integral tests for :func:`math.sqrt` function. + +.. + +.. bpo: 43921 +.. date: 2021-06-03-03-29-34 +.. nonce: nwH1FS +.. section: Tests + +Fix test_ssl.test_wrong_cert_tls13(): use ``suppress_ragged_eofs=False``, +since ``read()`` can raise :exc:`ssl.SSLEOFError` on Windows. Patch by +Victor Stinner. + +.. + +.. bpo: 43921 +.. date: 2021-06-02-17-41-42 +.. nonce: xP7yZ4 +.. section: Tests + +Fix test_pha_required_nocert() of test_ssl: catch two more EOF cases (when +the ``recv()`` method returns an empty string). Patch by Victor Stinner. + +.. + +.. bpo: 44131 +.. date: 2021-05-14-14-13-44 +.. nonce: YPizoC +.. section: Tests + +Add test_frozenmain to test_embed to test the :c:func:`Py_FrozenMain` C +function. Patch by Victor Stinner. + +.. + +.. bpo: 31904 +.. date: 2021-05-07-15-46-04 +.. nonce: 8dk3la +.. section: Tests + +Ignore error string case in test_file_not_exists(). + +.. + +.. bpo: 42083 +.. date: 2021-05-04-18-10-57 +.. nonce: EMS2TK +.. section: Tests + +Add test to check that ``PyStructSequence_NewType`` accepts a +``PyStructSequence_Desc`` with ``doc`` field set to ``NULL``. + +.. + +.. bpo: 35753 +.. date: 2020-10-25-19-20-26 +.. nonce: 2LT-hO +.. section: Tests + +Fix crash in doctest when doctest parses modules that include unwrappable +functions by skipping those functions. + +.. + +.. bpo: 30256 +.. date: 2019-09-25-18-10-10 +.. nonce: A5i76Q +.. section: Tests + +Add test for nested queues when using ``multiprocessing`` shared objects +``AutoProxy[Queue]`` inside ``ListProxy`` and ``DictProxy`` + +.. + +.. bpo: 45220 +.. date: 2021-09-16-18-00-43 +.. nonce: TgbkvW +.. section: Build + +Avoid building with the Windows 11 SDK previews automatically. This may be +overridden by setting the ``DefaultWindowsSDKVersion`` environment variable +before building. + +.. + +.. bpo: 45020 +.. date: 2021-09-14-10-07-23 +.. nonce: _VGGPv +.. section: Build + +Freeze stdlib modules that are imported during startup. This provides +significant performance improvements to startup. If necessary, use the +previously added "-X frozen_modules=off" commandline option to force +importing the source modules. + +.. + +.. bpo: 45188 +.. date: 2021-09-14-00-47-57 +.. nonce: MNbo_T +.. section: Build + +Windows builds now regenerate frozen modules as the first part of the build. +Previously the regeneration was later in the build, which would require it +to be restarted if any modules had changed. + +.. + +.. bpo: 45163 +.. date: 2021-09-11-06-05-23 +.. nonce: q7xT93 +.. section: Build + +Fixes Haiku platform build. + +.. + +.. bpo: 45067 +.. date: 2021-09-09-16-45-26 +.. nonce: mFmY92 +.. section: Build + +The ncurses function extended_color_content was introduced in 2017 + +(https://invisible-island.net/ncurses/NEWS.html#index-t20170401). The + +ncurses-devel package in CentOS 7 had a older version ncurses resulted in +compilation error. For compiling ncurses with extended color support, we +verify the version of the ncurses library >= 20170401. + +.. + +.. bpo: 45019 +.. date: 2021-08-26-13-10-46 +.. nonce: e0mo49 +.. section: Build + +Generate lines in relevant files for frozen modules. Up until now each of +the files had to be edited manually. This change makes it easier to add to +and modify the frozen modules. + +.. + +.. bpo: 44340 +.. date: 2021-07-19-01-09-56 +.. nonce: JNeOf4 +.. section: Build + +Add support for building with clang thin lto via --with-lto=thin/full. Patch +by Dong-hee Na and Brett Holman. + +.. + +.. bpo: 44535 +.. date: 2021-06-30-02-32-46 +.. nonce: M9iN4- +.. section: Build + +Enable building using a Visual Studio 2022 install on Windows. + +.. + +.. bpo: 43298 +.. date: 2021-06-19-11-50-03 +.. nonce: 9ircMb +.. section: Build + +Improved error message when building without a Windows SDK installed. + +.. + +.. bpo: 44381 +.. date: 2021-06-10-18-08-44 +.. nonce: Xpc1iX +.. section: Build + +The Windows build now accepts :envvar:`EnableControlFlowGuard` set to +``guard`` to enable CFG. + +.. + +.. bpo: 41282 +.. date: 2021-05-24-03-31-17 +.. nonce: L8nP44 +.. section: Build + +Fix broken ``make install`` that caused standard library extension modules +to be unnecessarily and incorrectly rebuilt during the install phase of +cpython. + +.. + +.. bpo: 45375 +.. date: 2021-10-05-12-41-53 +.. nonce: CohPP- +.. section: Windows + +Fixes an assertion failure due to searching for the standard library in +unnormalised paths. + +.. + +.. bpo: 45022 +.. date: 2021-09-03-18-05-21 +.. nonce: bgpD_r +.. section: Windows + +Update Windows release to include libffi 3.4.2 + +.. + +.. bpo: 45007 +.. date: 2021-08-27-23-50-02 +.. nonce: NIBlVG +.. section: Windows + +Update to OpenSSL 1.1.1l in Windows build + +.. + +.. bpo: 44848 +.. date: 2021-08-06-10-11-07 +.. nonce: ib3Jcz +.. section: Windows + +Upgrade Windows installer to use SQLite 3.36.0. + +.. + +.. bpo: 44572 +.. date: 2021-07-13-15-32-49 +.. nonce: gXvhDc +.. section: Windows + +Avoid consuming standard input in the :mod:`platform` module + +.. + +.. bpo: 44582 +.. date: 2021-07-07-21-07-18 +.. nonce: 4Mm6Hh +.. section: Windows + +Accelerate speed of :mod:`mimetypes` initialization using a native +implementation of the registry scan. + +.. + +.. bpo: 41299 +.. date: 2021-06-06-16-36-13 +.. nonce: Rg-vb_ +.. section: Windows + +Fix 16 milliseconds jitter when using timeouts in :mod:`threading`, such as +with :meth:`threading.Lock.acquire` or :meth:`threading.Condition.wait`. + +.. + +.. bpo: 42686 +.. date: 2021-01-01-21-21-03 +.. nonce: G_f-TC +.. section: Windows + +Build :mod:`sqlite3` with math functions enabled. Patch by Erlend E. +Aasland. + +.. + +.. bpo: 40263 +.. date: 2020-04-13-15-20-28 +.. nonce: 1KKEbu +.. section: Windows + +This is a follow-on bug from https://bugs.python.org/issue26903. Once that +is applied we run into an off-by-one assertion problem. The assert was not +correct. + +.. + +.. bpo: 45007 +.. date: 2021-08-30-00-04-10 +.. nonce: pixqUB +.. section: macOS + +Update macOS installer builds to use OpenSSL 1.1.1l. + +.. + +.. bpo: 34602 +.. date: 2021-08-27-16-55-10 +.. nonce: ZjHsYJ +.. section: macOS + +When building CPython on macOS with ``./configure +--with-undefined-behavior-sanitizer --with-pydebug``, the stack size is now +quadrupled to allow for the entire test suite to pass. + +.. + +.. bpo: 44848 +.. date: 2021-08-06-10-08-41 +.. nonce: 0uYXsE +.. section: macOS + +Update macOS installer to use SQLite 3.36.0. + +.. + +.. bpo: 44689 +.. date: 2021-07-20-22-27-01 +.. nonce: mmT_xH +.. section: macOS + +:meth:`ctypes.util.find_library` now works correctly on macOS 11 Big Sur +even if Python is built on an older version of macOS. Previously, when +built on older macOS systems, ``find_library`` was not able to find macOS +system libraries when running on Big Sur due to changes in how system +libraries are stored. + +.. + +.. bpo: 41972 +.. date: 2021-07-12-15-42-02 +.. nonce: yUjE8j +.. section: macOS + +The framework build's user header path in sysconfig is changed to add a +'pythonX.Y' component to match distutils's behavior. + +.. + +.. bpo: 43109 +.. date: 2021-05-24-21-15-41 +.. nonce: npKJ9c +.. section: macOS + +Allow --with-lto configure option to work with Apple-supplied Xcode or +Command Line Tools. + +.. + +.. bpo: 34932 +.. date: 2021-03-29-21-11-23 +.. nonce: f3Hdyd +.. section: macOS + +Add socket.TCP_KEEPALIVE support for macOS. Patch by Shane Harvey. + +.. + +.. bpo: 45296 +.. date: 2021-09-27-01-21-59 +.. nonce: 9H8rdY +.. section: IDLE + +On Windows, change exit/quit message to suggest Ctrl-D, which works, instead +of , which does not work in IDLE. + +.. + +.. bpo: 45193 +.. date: 2021-09-15-03-20-06 +.. nonce: G61_GV +.. section: IDLE + +Make completion boxes appear on Ubuntu again. + +.. + +.. bpo: 40128 +.. date: 2021-06-11-17-43-39 +.. nonce: 7vDN3U +.. section: IDLE + +Mostly fix completions on macOS when not using tcl/tk 8.6.11 (as with 3.9). +The added update_idletask call should be harmless and possibly helpful +otherwise. + +.. + +.. bpo: 33962 +.. date: 2021-06-10-00-50-02 +.. nonce: ikAUNg +.. section: IDLE + +Move the indent space setting from the Font tab to the new Windows tab. +Patch by Mark Roseman and Terry Jan Reedy. + +.. + +.. bpo: 40468 +.. date: 2021-06-08-03-04-51 +.. nonce: tUCGUb +.. section: IDLE + +Split the settings dialog General tab into Windows and Shell/ED tabs. Move +help sources, which extend the Help menu, to the Extensions tab. Make space +for new options and shorten the dialog. The latter makes the dialog better +fit small screens. + +.. + +.. bpo: 41611 +.. date: 2021-05-27-18-22-46 +.. nonce: jOKpfc +.. section: IDLE + +Avoid uncaught exceptions in ``AutoCompleteWindow.winconfig_event()``. + +.. + +.. bpo: 41611 +.. date: 2021-05-27-13-39-43 +.. nonce: liNQqj +.. section: IDLE + +Fix IDLE sometimes freezing upon tab-completion on macOS. + +.. + +.. bpo: 44010 +.. date: 2021-05-09-09-02-09 +.. nonce: TaLe9x +.. section: IDLE + +Highlight the new :ref:`match ` statement's :ref:`soft keywords +`: :keyword:`match`, :keyword:`case `, and :keyword:`_ +`. However, this highlighting is not perfect and will be +incorrect in some rare cases, including some ``_``-s in ``case`` patterns. + +.. + +.. bpo: 44026 +.. date: 2021-05-05-09-45-24 +.. nonce: m2Z0zR +.. section: IDLE + +Include interpreter's typo fix suggestions in message line for NameErrors +and AttributeErrors. Patch by E. Paine. + +.. + +.. bpo: 44786 +.. date: 2021-09-14-11-44-26 +.. nonce: DU0LC0 +.. section: Tools/Demos + +Fix a warning in regular expression in the c-analyzer script. + +.. + +.. bpo: 44967 +.. date: 2021-08-26-11-57-31 +.. nonce: UT1RMV +.. section: Tools/Demos + +pydoc now returns a non-zero status code when a module cannot be found. + +.. + +.. bpo: 44978 +.. date: 2021-08-22-11-45-31 +.. nonce: QupKV3 +.. section: Tools/Demos + +Allow the Argument Clinic tool to handle ``__complex__`` special methods. + +.. + +.. bpo: 43425 +.. date: 2021-07-01-22-21-25 +.. nonce: t65len +.. section: Tools/Demos + +Removed the 'test2to3' demo project that demonstrated using lib2to3 to +support Python 2.x and Python 3.x from a single source in a distutils +package. Patch by Dong-hee Na + +.. + +.. bpo: 44074 +.. date: 2021-05-08-13-57-00 +.. nonce: F09kCK +.. section: Tools/Demos + +Make patchcheck automatically detect the correct base branch name +(previously it was hardcoded to 'master') + +.. + +.. bpo: 20291 +.. date: 2020-02-25-18-22-09 +.. nonce: AyrDiZ +.. section: Tools/Demos + +Added support for variadic positional parameters in Argument Clinic. + +.. + +.. bpo: 41710 +.. date: 2021-09-30-03-14-35 +.. nonce: DDWJKx +.. section: C API + +The PyThread_acquire_lock_timed() function now clamps the timeout if it is +too large, rather than aborting the process. Patch by Victor Stinner. + +.. + +.. bpo: 44687 +.. date: 2021-09-19-17-18-25 +.. nonce: 3fqDRC +.. section: C API + +:meth:`BufferedReader.peek` no longer raises :exc:`ValueError` when the +entire file has already been buffered. + +.. + +.. bpo: 45116 +.. date: 2021-09-16-18-05-20 +.. nonce: WxXewl +.. section: C API + +Add the :c:macro:`Py_ALWAYS_INLINE` macro to ask the compiler to always +inline a static inline function. The compiler can ignore it and decides to +not inline the function. Patch by Victor Stinner. + +.. + +.. bpo: 45094 +.. date: 2021-09-03-15-53-43 +.. nonce: tinXwL +.. section: C API + +Add the :c:macro:`Py_NO_INLINE` macro to disable inlining on a function. +Patch by Victor Stinner. + +.. + +.. bpo: 45061 +.. date: 2021-08-31-15-21-36 +.. nonce: ZH0HVe +.. section: C API + +Add a deallocator to the :class:`bool` type to detect refcount bugs in C +extensions which call ``Py_DECREF(Py_True);`` or ``Py_DECREF(Py_False);`` by +mistake. Patch by Victor Stinner. + +.. + +.. bpo: 42035 +.. date: 2021-08-02-20-49-36 +.. nonce: HTBcZt +.. section: C API + +Add a new :c:func:`PyType_GetQualName` function to get type's qualified +name. + +.. + +.. bpo: 41103 +.. date: 2021-07-29-16-04-28 +.. nonce: hiKKcF +.. section: C API + +Reverts removal of the old buffer protocol because they are part of stable +ABI. + +.. + +.. bpo: 44751 +.. date: 2021-07-27-17-29-12 +.. nonce: 4qmbDG +.. section: C API + +Remove ``crypt.h`` include from the public ``Python.h`` header. + +.. + +.. bpo: 42747 +.. date: 2021-07-20-16-21-06 +.. nonce: rRxjUY +.. section: C API + +The ``Py_TPFLAGS_HAVE_VERSION_TAG`` type flag now does nothing. The +``Py_TPFLAGS_HAVE_AM_SEND`` flag (which was added in 3.10) is removed. Both +were unnecessary because it is not possible to have type objects with the +relevant fields missing. + +.. + +.. bpo: 44530 +.. date: 2021-06-28-23-44-47 +.. nonce: qij7YC +.. section: C API + +Added the ``co_qualname`` to the ``PyCodeObject`` structure to propagate the +qualified name from the compiler to code objects. + +Patch by Gabriele N. Tornetta + +.. + +.. bpo: 44441 +.. date: 2021-06-23-12-12-04 +.. nonce: 3p14JB +.. section: C API + +:c:func:`Py_RunMain` now resets :c:data:`PyImport_Inittab` to its initial +value at exit. It must be possible to call :c:func:`PyImport_AppendInittab` +or :c:func:`PyImport_ExtendInittab` at each Python initialization. Patch by +Victor Stinner. + +.. + +.. bpo: 39947 +.. date: 2021-06-23-10-31-45 +.. nonce: je_HMo +.. section: C API + +Remove 4 private trashcan C API functions which were only kept for the +backward compatibility of the stable ABI with Python 3.8 and older, since +the trashcan API was not usable with the limited C API on Python 3.8 and +older. The trashcan API was excluded from the limited C API in Python 3.9. + +Removed functions: + +* _PyTrash_deposit_object() +* _PyTrash_destroy_chain() +* _PyTrash_thread_deposit_object() +* _PyTrash_thread_destroy_chain() + +The trashcan C API was never usable with the limited C API, since old +trashcan macros accessed directly :c:type:`PyThreadState` members like +``_tstate->trash_delete_nesting``, whereas the :c:type:`PyThreadState` +structure is opaque in the limited C API. + +Exclude also the the ``PyTrash_UNWIND_LEVEL`` constant from the C API. + +Patch by Victor Stinner. + +.. + +.. bpo: 40939 +.. date: 2021-06-22-17-00-06 +.. nonce: CGB0I5 +.. section: C API + +Removed documentation for the removed ``PyParser_*`` C API. + +.. + +.. bpo: 43795 +.. date: 2021-06-15-16-28-18 +.. nonce: fy0AXK +.. section: C API + +The list in :ref:`stable-abi-list` now shows the public name +:c:struct:`PyFrameObject` rather than ``_frame``. The non-existing entry +``_node`` no longer appears in the list. + +.. + +.. bpo: 44378 +.. date: 2021-06-10-15-22-31 +.. nonce: jGYakF +.. section: C API + +:c:func:`Py_IS_TYPE` no longer uses :c:func:`Py_TYPE` to avoid a compiler +warning: no longer cast ``const PyObject*`` to ``PyObject*``. Patch by +Victor Stinner. + +.. + +.. bpo: 39573 +.. date: 2021-06-03-00-59-48 +.. nonce: -elHTJ +.. section: C API + +Convert the :c:func:`Py_TYPE` and :c:func:`Py_SIZE` macros to static inline +functions. The :c:func:`Py_SET_TYPE` and :c:func:`Py_SET_SIZE` functions +must now be used to set an object type and size. Patch by Victor Stinner. + +.. + +.. bpo: 44263 +.. date: 2021-05-31-11-31-13 +.. nonce: 8mIOfV +.. section: C API + +The :c:func:`PyType_Ready` function now raises an error if a type is defined +with the :const:`Py_TPFLAGS_HAVE_GC` flag set but has no traverse function +(:c:member:`PyTypeObject.tp_traverse`). Patch by Victor Stinner. + +.. + +.. bpo: 43795 +.. date: 2021-05-19-15-09-47 +.. nonce: WAHRxt +.. section: C API + +The undocumented function :c:func:`Py_FrozenMain` is removed from the +Limited API. + +.. + +.. bpo: 44113 +.. date: 2021-05-12-12-24-45 +.. nonce: DcgOqE +.. section: C API + +Deprecate the following functions to configure the Python initialization: + +* :c:func:`PySys_AddWarnOptionUnicode` +* :c:func:`PySys_AddWarnOption` +* :c:func:`PySys_AddXOption` +* :c:func:`PySys_HasWarnOptions` +* :c:func:`Py_SetPath` +* :c:func:`Py_SetProgramName` +* :c:func:`Py_SetPythonHome` +* :c:func:`Py_SetStandardStreamEncoding` +* :c:func:`_Py_SetProgramFullPath` + +Use the new :c:type:`PyConfig` API of the :ref:`Python Initialization +Configuration ` instead (:pep:`587`). + +.. + +.. bpo: 44094 +.. date: 2021-05-10-14-34-22 +.. nonce: HayXZO +.. section: C API + +Remove ``PyErr_SetFromErrnoWithUnicodeFilename()``, +``PyErr_SetFromWindowsErrWithUnicodeFilename()``, and +``PyErr_SetExcFromWindowsErrWithUnicodeFilename()``. They are not documented +and have been deprecated since Python 3.3. + +.. + +.. bpo: 43795 +.. date: 2021-05-05-19-04-50 +.. nonce: 9Ojj73 +.. section: C API + +:c:func:`PyCodec_Unregister` is now properly exported as a function in the +Windows Stable ABI DLL. + +.. + +.. bpo: 44029 +.. date: 2021-05-04-17-43-39 +.. nonce: ayX4PR +.. section: C API + +Remove deprecated ``Py_UNICODE`` APIs: ``PyUnicode_Encode``, +``PyUnicode_EncodeUTF7``, ``PyUnicode_EncodeUTF8``, +``PyUnicode_EncodeUTF16``, ``PyUnicode_EncodeUTF32``, +``PyUnicode_EncodeLatin1``, ``PyUnicode_EncodeMBCS``, +``PyUnicode_EncodeDecimal``, ``PyUnicode_EncodeRawUnicodeEscape``, +``PyUnicode_EncodeCharmap``, ``PyUnicode_EncodeUnicodeEscape``, +``PyUnicode_TransformDecimalToASCII``, ``PyUnicode_TranslateCharmap``, +``PyUnicodeEncodeError_Create``, ``PyUnicodeTranslateError_Create``. See +:pep:`393` and :pep:`624` for reference. + +.. + +.. bpo: 42035 +.. date: 2020-12-23-01-28-50 +.. nonce: S9eUm0 +.. section: C API + +Add a new :c:func:`PyType_GetName` function to get type's short name. diff --git a/Misc/NEWS.d/next/Build/2021-05-24-03-31-17.bpo-41282.L8nP44.rst b/Misc/NEWS.d/next/Build/2021-05-24-03-31-17.bpo-41282.L8nP44.rst deleted file mode 100644 index cc6eadefc6cba..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-05-24-03-31-17.bpo-41282.L8nP44.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix broken ``make install`` that caused standard library extension modules -to be unnecessarily and incorrectly rebuilt during the install phase of -cpython. diff --git a/Misc/NEWS.d/next/Build/2021-06-10-18-08-44.bpo-44381.Xpc1iX.rst b/Misc/NEWS.d/next/Build/2021-06-10-18-08-44.bpo-44381.Xpc1iX.rst deleted file mode 100644 index 002112c4b5567..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-06-10-18-08-44.bpo-44381.Xpc1iX.rst +++ /dev/null @@ -1,2 +0,0 @@ -The Windows build now accepts :envvar:`EnableControlFlowGuard` set to -``guard`` to enable CFG. diff --git a/Misc/NEWS.d/next/Build/2021-06-19-11-50-03.bpo-43298.9ircMb.rst b/Misc/NEWS.d/next/Build/2021-06-19-11-50-03.bpo-43298.9ircMb.rst deleted file mode 100644 index 3bdc24b147a3e..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-06-19-11-50-03.bpo-43298.9ircMb.rst +++ /dev/null @@ -1 +0,0 @@ -Improved error message when building without a Windows SDK installed. diff --git a/Misc/NEWS.d/next/Build/2021-06-30-02-32-46.bpo-44535.M9iN4-.rst b/Misc/NEWS.d/next/Build/2021-06-30-02-32-46.bpo-44535.M9iN4-.rst deleted file mode 100644 index e06d0d304852f..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-06-30-02-32-46.bpo-44535.M9iN4-.rst +++ /dev/null @@ -1 +0,0 @@ -Enable building using a Visual Studio 2022 install on Windows. diff --git a/Misc/NEWS.d/next/Build/2021-07-19-01-09-56.bpo-44340.JNeOf4.rst b/Misc/NEWS.d/next/Build/2021-07-19-01-09-56.bpo-44340.JNeOf4.rst deleted file mode 100644 index cf19eb6052e8d..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-07-19-01-09-56.bpo-44340.JNeOf4.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add support for building with clang thin lto via --with-lto=thin/full. Patch -by Dong-hee Na and Brett Holman. diff --git a/Misc/NEWS.d/next/Build/2021-08-26-13-10-46.bpo-45019.e0mo49.rst b/Misc/NEWS.d/next/Build/2021-08-26-13-10-46.bpo-45019.e0mo49.rst deleted file mode 100644 index d11c6451462bd..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-08-26-13-10-46.bpo-45019.e0mo49.rst +++ /dev/null @@ -1,3 +0,0 @@ -Generate lines in relevant files for frozen modules. Up until now each of -the files had to be edited manually. This change makes it easier to add to -and modify the frozen modules. diff --git a/Misc/NEWS.d/next/Build/2021-09-09-16-45-26.bpo-45067.mFmY92.rst b/Misc/NEWS.d/next/Build/2021-09-09-16-45-26.bpo-45067.mFmY92.rst deleted file mode 100644 index a89736eb33e82..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-09-09-16-45-26.bpo-45067.mFmY92.rst +++ /dev/null @@ -1,7 +0,0 @@ -The ncurses function extended_color_content was introduced in 2017 - -(https://invisible-island.net/ncurses/NEWS.html#index-t20170401). The - -ncurses-devel package in CentOS 7 had a older version ncurses resulted in -compilation error. For compiling ncurses with extended color support, we -verify the version of the ncurses library >= 20170401. diff --git a/Misc/NEWS.d/next/Build/2021-09-11-06-05-23.bpo-45163.q7xT93.rst b/Misc/NEWS.d/next/Build/2021-09-11-06-05-23.bpo-45163.q7xT93.rst deleted file mode 100644 index 2b656bcbc230b..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-09-11-06-05-23.bpo-45163.q7xT93.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes Haiku platform build. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Build/2021-09-14-00-47-57.bpo-45188.MNbo_T.rst b/Misc/NEWS.d/next/Build/2021-09-14-00-47-57.bpo-45188.MNbo_T.rst deleted file mode 100644 index df470e8eeb30c..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-09-14-00-47-57.bpo-45188.MNbo_T.rst +++ /dev/null @@ -1,3 +0,0 @@ -Windows builds now regenerate frozen modules as the first part of the build. -Previously the regeneration was later in the build, which would require it -to be restarted if any modules had changed. diff --git a/Misc/NEWS.d/next/Build/2021-09-14-10-07-23.bpo-45020._VGGPv.rst b/Misc/NEWS.d/next/Build/2021-09-14-10-07-23.bpo-45020._VGGPv.rst deleted file mode 100644 index 67d61c3f88040..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-09-14-10-07-23.bpo-45020._VGGPv.rst +++ /dev/null @@ -1,4 +0,0 @@ -Freeze stdlib modules that are imported during startup. This provides -significant performance improvements to startup. If necessary, use the -previously added "-X frozen_modules=off" commandline option to force -importing the source modules. diff --git a/Misc/NEWS.d/next/Build/2021-09-16-18-00-43.bpo-45220.TgbkvW.rst b/Misc/NEWS.d/next/Build/2021-09-16-18-00-43.bpo-45220.TgbkvW.rst deleted file mode 100644 index 8bbd634fa61a3..0000000000000 --- a/Misc/NEWS.d/next/Build/2021-09-16-18-00-43.bpo-45220.TgbkvW.rst +++ /dev/null @@ -1,3 +0,0 @@ -Avoid building with the Windows 11 SDK previews automatically. This may be -overridden by setting the ``DefaultWindowsSDKVersion`` environment variable -before building. diff --git a/Misc/NEWS.d/next/C API/2020-12-23-01-28-50.bpo-42035.S9eUm0.rst b/Misc/NEWS.d/next/C API/2020-12-23-01-28-50.bpo-42035.S9eUm0.rst deleted file mode 100644 index 8adb20e62d1c4..0000000000000 --- a/Misc/NEWS.d/next/C API/2020-12-23-01-28-50.bpo-42035.S9eUm0.rst +++ /dev/null @@ -1 +0,0 @@ -Add a new :c:func:`PyType_GetName` function to get type's short name. diff --git a/Misc/NEWS.d/next/C API/2021-05-04-17-43-39.bpo-44029.ayX4PR.rst b/Misc/NEWS.d/next/C API/2021-05-04-17-43-39.bpo-44029.ayX4PR.rst deleted file mode 100644 index cf55e41bf332c..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-04-17-43-39.bpo-44029.ayX4PR.rst +++ /dev/null @@ -1,9 +0,0 @@ -Remove deprecated ``Py_UNICODE`` APIs: ``PyUnicode_Encode``, -``PyUnicode_EncodeUTF7``, ``PyUnicode_EncodeUTF8``, -``PyUnicode_EncodeUTF16``, ``PyUnicode_EncodeUTF32``, -``PyUnicode_EncodeLatin1``, ``PyUnicode_EncodeMBCS``, -``PyUnicode_EncodeDecimal``, ``PyUnicode_EncodeRawUnicodeEscape``, -``PyUnicode_EncodeCharmap``, ``PyUnicode_EncodeUnicodeEscape``, -``PyUnicode_TransformDecimalToASCII``, ``PyUnicode_TranslateCharmap``, -``PyUnicodeEncodeError_Create``, ``PyUnicodeTranslateError_Create``. See -:pep:`393` and :pep:`624` for reference. diff --git a/Misc/NEWS.d/next/C API/2021-05-05-19-04-50.bpo-43795.9Ojj73.rst b/Misc/NEWS.d/next/C API/2021-05-05-19-04-50.bpo-43795.9Ojj73.rst deleted file mode 100644 index 20a3823f1f01c..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-05-19-04-50.bpo-43795.9Ojj73.rst +++ /dev/null @@ -1,2 +0,0 @@ -:c:func:`PyCodec_Unregister` is now properly exported as a function in the -Windows Stable ABI DLL. diff --git a/Misc/NEWS.d/next/C API/2021-05-10-14-34-22.bpo-44094.HayXZO.rst b/Misc/NEWS.d/next/C API/2021-05-10-14-34-22.bpo-44094.HayXZO.rst deleted file mode 100644 index eea9e0bf28208..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-10-14-34-22.bpo-44094.HayXZO.rst +++ /dev/null @@ -1,4 +0,0 @@ -Remove ``PyErr_SetFromErrnoWithUnicodeFilename()``, -``PyErr_SetFromWindowsErrWithUnicodeFilename()``, and -``PyErr_SetExcFromWindowsErrWithUnicodeFilename()``. They are not documented -and have been deprecated since Python 3.3. diff --git a/Misc/NEWS.d/next/C API/2021-05-12-12-24-45.bpo-44113.DcgOqE.rst b/Misc/NEWS.d/next/C API/2021-05-12-12-24-45.bpo-44113.DcgOqE.rst deleted file mode 100644 index 45f67efa10573..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-12-12-24-45.bpo-44113.DcgOqE.rst +++ /dev/null @@ -1,14 +0,0 @@ -Deprecate the following functions to configure the Python initialization: - -* :c:func:`PySys_AddWarnOptionUnicode` -* :c:func:`PySys_AddWarnOption` -* :c:func:`PySys_AddXOption` -* :c:func:`PySys_HasWarnOptions` -* :c:func:`Py_SetPath` -* :c:func:`Py_SetProgramName` -* :c:func:`Py_SetPythonHome` -* :c:func:`Py_SetStandardStreamEncoding` -* :c:func:`_Py_SetProgramFullPath` - -Use the new :c:type:`PyConfig` API of the :ref:`Python Initialization -Configuration ` instead (:pep:`587`). diff --git a/Misc/NEWS.d/next/C API/2021-05-19-15-09-47.bpo-43795.WAHRxt.rst b/Misc/NEWS.d/next/C API/2021-05-19-15-09-47.bpo-43795.WAHRxt.rst deleted file mode 100644 index 23db2330ac396..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-19-15-09-47.bpo-43795.WAHRxt.rst +++ /dev/null @@ -1 +0,0 @@ -The undocumented function :c:func:`Py_FrozenMain` is removed from the Limited API. diff --git a/Misc/NEWS.d/next/C API/2021-05-31-11-31-13.bpo-44263.8mIOfV.rst b/Misc/NEWS.d/next/C API/2021-05-31-11-31-13.bpo-44263.8mIOfV.rst deleted file mode 100644 index aa831a2083c48..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-05-31-11-31-13.bpo-44263.8mIOfV.rst +++ /dev/null @@ -1,4 +0,0 @@ -The :c:func:`PyType_Ready` function now raises an error if a type is defined -with the :const:`Py_TPFLAGS_HAVE_GC` flag set but has no traverse function -(:c:member:`PyTypeObject.tp_traverse`). -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-06-03-00-59-48.bpo-39573.-elHTJ.rst b/Misc/NEWS.d/next/C API/2021-06-03-00-59-48.bpo-39573.-elHTJ.rst deleted file mode 100644 index d9641ed97e170..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-03-00-59-48.bpo-39573.-elHTJ.rst +++ /dev/null @@ -1,3 +0,0 @@ -Convert the :c:func:`Py_TYPE` and :c:func:`Py_SIZE` macros to static inline -functions. The :c:func:`Py_SET_TYPE` and :c:func:`Py_SET_SIZE` functions -must now be used to set an object type and size. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-06-10-15-22-31.bpo-44378.jGYakF.rst b/Misc/NEWS.d/next/C API/2021-06-10-15-22-31.bpo-44378.jGYakF.rst deleted file mode 100644 index b620b499f2351..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-10-15-22-31.bpo-44378.jGYakF.rst +++ /dev/null @@ -1,3 +0,0 @@ -:c:func:`Py_IS_TYPE` no longer uses :c:func:`Py_TYPE` to avoid a compiler -warning: no longer cast ``const PyObject*`` to ``PyObject*``. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-06-15-16-28-18.bpo-43795.fy0AXK.rst b/Misc/NEWS.d/next/C API/2021-06-15-16-28-18.bpo-43795.fy0AXK.rst deleted file mode 100644 index 8d029a0457908..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-15-16-28-18.bpo-43795.fy0AXK.rst +++ /dev/null @@ -1,3 +0,0 @@ -The list in :ref:`stable-abi-list` now shows the public name -:c:struct:`PyFrameObject` rather than ``_frame``. The non-existing -entry ``_node`` no longer appears in the list. diff --git a/Misc/NEWS.d/next/C API/2021-06-22-17-00-06.bpo-40939.CGB0I5.rst b/Misc/NEWS.d/next/C API/2021-06-22-17-00-06.bpo-40939.CGB0I5.rst deleted file mode 100644 index 2531ac1ba3c12..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-22-17-00-06.bpo-40939.CGB0I5.rst +++ /dev/null @@ -1 +0,0 @@ -Removed documentation for the removed ``PyParser_*`` C API. diff --git a/Misc/NEWS.d/next/C API/2021-06-23-10-31-45.bpo-39947.je_HMo.rst b/Misc/NEWS.d/next/C API/2021-06-23-10-31-45.bpo-39947.je_HMo.rst deleted file mode 100644 index 43adbffc7cce2..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-23-10-31-45.bpo-39947.je_HMo.rst +++ /dev/null @@ -1,20 +0,0 @@ -Remove 4 private trashcan C API functions which were only kept for the backward -compatibility of the stable ABI with Python 3.8 and older, since the trashcan -API was not usable with the limited C API on Python 3.8 and older. The -trashcan API was excluded from the limited C API in Python 3.9. - -Removed functions: - -* _PyTrash_deposit_object() -* _PyTrash_destroy_chain() -* _PyTrash_thread_deposit_object() -* _PyTrash_thread_destroy_chain() - -The trashcan C API was never usable with the limited C API, since old trashcan -macros accessed directly :c:type:`PyThreadState` members like -``_tstate->trash_delete_nesting``, whereas the :c:type:`PyThreadState` -structure is opaque in the limited C API. - -Exclude also the the ``PyTrash_UNWIND_LEVEL`` constant from the C API. - -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-06-23-12-12-04.bpo-44441.3p14JB.rst b/Misc/NEWS.d/next/C API/2021-06-23-12-12-04.bpo-44441.3p14JB.rst deleted file mode 100644 index 80c4282c18ea6..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-23-12-12-04.bpo-44441.3p14JB.rst +++ /dev/null @@ -1,4 +0,0 @@ -:c:func:`Py_RunMain` now resets :c:data:`PyImport_Inittab` to its initial value -at exit. It must be possible to call :c:func:`PyImport_AppendInittab` or -:c:func:`PyImport_ExtendInittab` at each Python initialization. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-06-28-23-44-47.bpo-44530.qij7YC.rst b/Misc/NEWS.d/next/C API/2021-06-28-23-44-47.bpo-44530.qij7YC.rst deleted file mode 100644 index 6200f9b97d7fc..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-06-28-23-44-47.bpo-44530.qij7YC.rst +++ /dev/null @@ -1,4 +0,0 @@ -Added the ``co_qualname`` to the ``PyCodeObject`` structure to propagate the -qualified name from the compiler to code objects. - -Patch by Gabriele N. Tornetta diff --git a/Misc/NEWS.d/next/C API/2021-07-20-16-21-06.bpo-42747.rRxjUY.rst b/Misc/NEWS.d/next/C API/2021-07-20-16-21-06.bpo-42747.rRxjUY.rst deleted file mode 100644 index c7ac5a776e2ed..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-07-20-16-21-06.bpo-42747.rRxjUY.rst +++ /dev/null @@ -1,4 +0,0 @@ -The ``Py_TPFLAGS_HAVE_VERSION_TAG`` type flag now does nothing. The -``Py_TPFLAGS_HAVE_AM_SEND`` flag (which was added in 3.10) is removed. Both -were unnecessary because it is not possible to have type objects with the -relevant fields missing. diff --git a/Misc/NEWS.d/next/C API/2021-07-27-17-29-12.bpo-44751.4qmbDG.rst b/Misc/NEWS.d/next/C API/2021-07-27-17-29-12.bpo-44751.4qmbDG.rst deleted file mode 100644 index d7b9f09819669..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-07-27-17-29-12.bpo-44751.4qmbDG.rst +++ /dev/null @@ -1 +0,0 @@ -Remove ``crypt.h`` include from the public ``Python.h`` header. diff --git a/Misc/NEWS.d/next/C API/2021-07-29-16-04-28.bpo-41103.hiKKcF.rst b/Misc/NEWS.d/next/C API/2021-07-29-16-04-28.bpo-41103.hiKKcF.rst deleted file mode 100644 index af06654ba2040..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-07-29-16-04-28.bpo-41103.hiKKcF.rst +++ /dev/null @@ -1,2 +0,0 @@ -Reverts removal of the old buffer protocol because they are part of stable -ABI. diff --git a/Misc/NEWS.d/next/C API/2021-08-02-20-49-36.bpo-42035.HTBcZt.rst b/Misc/NEWS.d/next/C API/2021-08-02-20-49-36.bpo-42035.HTBcZt.rst deleted file mode 100644 index 4631c43fdd531..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-08-02-20-49-36.bpo-42035.HTBcZt.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add a new :c:func:`PyType_GetQualName` function to get type's qualified -name. diff --git a/Misc/NEWS.d/next/C API/2021-08-31-15-21-36.bpo-45061.ZH0HVe.rst b/Misc/NEWS.d/next/C API/2021-08-31-15-21-36.bpo-45061.ZH0HVe.rst deleted file mode 100644 index 58bd534601fb9..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-08-31-15-21-36.bpo-45061.ZH0HVe.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add a deallocator to the :class:`bool` type to detect refcount bugs in C -extensions which call ``Py_DECREF(Py_True);`` or ``Py_DECREF(Py_False);`` by -mistake. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-09-03-15-53-43.bpo-45094.tinXwL.rst b/Misc/NEWS.d/next/C API/2021-09-03-15-53-43.bpo-45094.tinXwL.rst deleted file mode 100644 index 84b01b23b435f..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-09-03-15-53-43.bpo-45094.tinXwL.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add the :c:macro:`Py_NO_INLINE` macro to disable inlining on a function. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-09-16-18-05-20.bpo-45116.WxXewl.rst b/Misc/NEWS.d/next/C API/2021-09-16-18-05-20.bpo-45116.WxXewl.rst deleted file mode 100644 index cf3db5ca2a023..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-09-16-18-05-20.bpo-45116.WxXewl.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add the :c:macro:`Py_ALWAYS_INLINE` macro to ask the compiler to always -inline a static inline function. The compiler can ignore it and decides to -not inline the function. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst b/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst deleted file mode 100644 index d38fa6057f6f9..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-09-19-17-18-25.bpo-44687.3fqDRC.rst +++ /dev/null @@ -1 +0,0 @@ -:meth:`BufferedReader.peek` no longer raises :exc:`ValueError` when the entire file has already been buffered. diff --git a/Misc/NEWS.d/next/C API/2021-09-30-03-14-35.bpo-41710.DDWJKx.rst b/Misc/NEWS.d/next/C API/2021-09-30-03-14-35.bpo-41710.DDWJKx.rst deleted file mode 100644 index 902c7cc8f2b99..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-09-30-03-14-35.bpo-41710.DDWJKx.rst +++ /dev/null @@ -1,2 +0,0 @@ -The PyThread_acquire_lock_timed() function now clamps the timeout if it is -too large, rather than aborting the process. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2018-05-11-12-44-03.bpo-33346.ZgBkvB.rst b/Misc/NEWS.d/next/Core and Builtins/2018-05-11-12-44-03.bpo-33346.ZgBkvB.rst deleted file mode 100644 index 9c91a8c0bf9ee..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2018-05-11-12-44-03.bpo-33346.ZgBkvB.rst +++ /dev/null @@ -1,3 +0,0 @@ -Asynchronous comprehensions are now allowed inside comprehensions in -asynchronous functions. Outer comprehensions implicitly become -asynchronous. diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-21-14-18-32.bpo-39091.dOexgQ.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-21-14-18-32.bpo-39091.dOexgQ.rst deleted file mode 100644 index c3b4e810d658b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2019-12-21-14-18-32.bpo-39091.dOexgQ.rst +++ /dev/null @@ -1 +0,0 @@ -Fix crash when using passing a non-exception to a generator's ``throw()`` method. Patch by Noah Oxer diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-06-02-13-21-14.bpo-11105.wceryW.rst b/Misc/NEWS.d/next/Core and Builtins/2020-06-02-13-21-14.bpo-11105.wceryW.rst deleted file mode 100644 index 8891936cd8871..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2020-06-02-13-21-14.bpo-11105.wceryW.rst +++ /dev/null @@ -1,3 +0,0 @@ -When compiling :class:`ast.AST` objects with recursive references -through :func:`compile`, the interpreter doesn't crash anymore instead -it raises a :exc:`RecursionError`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-01-13-19-34-41.bpo-28146.AZBBkH.rst b/Misc/NEWS.d/next/Core and Builtins/2021-01-13-19-34-41.bpo-28146.AZBBkH.rst deleted file mode 100644 index e619881938953..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-01-13-19-34-41.bpo-28146.AZBBkH.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a confusing error message in :func:`str.format`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-03-22-17-50-30.bpo-17792._zssjS.rst b/Misc/NEWS.d/next/Core and Builtins/2021-03-22-17-50-30.bpo-17792._zssjS.rst deleted file mode 100644 index 768cbf14efad9..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-03-22-17-50-30.bpo-17792._zssjS.rst +++ /dev/null @@ -1 +0,0 @@ -More accurate error messages for access of unbound locals or free vars. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-02-15-02-16.bpo-43693.l3Ureu.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-02-15-02-16.bpo-43693.l3Ureu.rst deleted file mode 100644 index 948c4d52482dc..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-04-02-15-02-16.bpo-43693.l3Ureu.rst +++ /dev/null @@ -1,4 +0,0 @@ -Compute cell offsets relative to locals in compiler. Allows the interpreter -to treats locals and cells a single array, which is slightly more efficient. -Also make the LOAD_CLOSURE opcode an alias for LOAD_FAST. Preserving -LOAD_CLOSURE helps keep bytecode a bit more readable. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-17-16-08-00.bpo-43879.zkyJgh.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-17-16-08-00.bpo-43879.zkyJgh.rst deleted file mode 100644 index 98b51736904a0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-04-17-16-08-00.bpo-43879.zkyJgh.rst +++ /dev/null @@ -1 +0,0 @@ -Add native_thread_id to PyThreadState. Patch by Gabriele N. Tornetta. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-18-18-07-33.bpo-43833.oChkCi.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-18-18-07-33.bpo-43833.oChkCi.rst deleted file mode 100644 index 2adbdba651b83..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-04-18-18-07-33.bpo-43833.oChkCi.rst +++ /dev/null @@ -1,4 +0,0 @@ -Emit a deprecation warning if the numeric literal is immediately followed by -one of keywords: and, else, for, if, in, is, or. Raise a syntax error with -more informative message if it is immediately followed by other keyword or -identifier. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-23-03-46-45.bpo-43918.nNDY3S.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-23-03-46-45.bpo-43918.nNDY3S.rst deleted file mode 100644 index f2f33f02abbd9..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-04-23-03-46-45.bpo-43918.nNDY3S.rst +++ /dev/null @@ -1 +0,0 @@ -Document the signature and ``default`` argument in the docstring of the new ``anext`` builtin. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-30-15-48-36.bpo-40222.j3VxeX.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-30-15-48-36.bpo-40222.j3VxeX.rst deleted file mode 100644 index b774475512278..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-04-30-15-48-36.bpo-40222.j3VxeX.rst +++ /dev/null @@ -1,7 +0,0 @@ -"Zero cost" exception handling. - -* Uses a lookup table to determine how to handle exceptions. -* Removes SETUP_FINALLY and POP_TOP block instructions, eliminating the runtime overhead of try statements. -* Reduces the size of the frame object by about 60%. - -Patch by Mark Shannon diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-04-01-01-04.bpo-43822.9VeCg0.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-04-01-01-04.bpo-43822.9VeCg0.rst deleted file mode 100644 index b8815cddf4e2c..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-04-01-01-04.bpo-43822.9VeCg0.rst +++ /dev/null @@ -1,2 +0,0 @@ -The parser will prioritize tokenizer errors over custom syntax errors when -raising exceptions. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-08-17-18-37.bpo-43149.Kp5FxD.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-08-17-18-37.bpo-43149.Kp5FxD.rst deleted file mode 100644 index cc1983ec3d464..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-08-17-18-37.bpo-43149.Kp5FxD.rst +++ /dev/null @@ -1,2 +0,0 @@ -Corrent the syntax error message regarding multiple exception types to not -refer to "exception groups". Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-08-19-54-57.bpo-28307.7ysaVW.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-08-19-54-57.bpo-28307.7ysaVW.rst deleted file mode 100644 index 86ac3254eaea8..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-08-19-54-57.bpo-28307.7ysaVW.rst +++ /dev/null @@ -1,3 +0,0 @@ -Compiler now optimizes simple C-style formatting with literal format -containing only format codes %s, %r and %a by converting them to f-string -expressions. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-10-18-49-13.bpo-26110.EQzqqA.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-10-18-49-13.bpo-26110.EQzqqA.rst deleted file mode 100644 index b5d9159000bad..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-10-18-49-13.bpo-26110.EQzqqA.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add ``CALL_METHOD_KW`` opcode to speed up method calls with keyword -arguments. Idea originated from PyPy. A side effect is executing -``CALL_METHOD`` is now branchless in the evaluation loop. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-11-21-52-44.bpo-44110.VqbAsB.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-11-21-52-44.bpo-44110.VqbAsB.rst deleted file mode 100644 index 3b82e425ab4a7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-11-21-52-44.bpo-44110.VqbAsB.rst +++ /dev/null @@ -1 +0,0 @@ -Improve :func:`str.__getitem__` error message diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-12-14-26-16.bpo-44114.p-WfAE.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-12-14-26-16.bpo-44114.p-WfAE.rst deleted file mode 100644 index c50b1594cae35..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-12-14-26-16.bpo-44114.p-WfAE.rst +++ /dev/null @@ -1 +0,0 @@ -Fix incorrect dictkeys_reversed and dictitems_reversed function signatures in C code, which broke webassembly builds. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-14-20-03-32.bpo-44032.OzT1ob.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-14-20-03-32.bpo-44032.OzT1ob.rst deleted file mode 100644 index fd2dec80cddf1..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-14-20-03-32.bpo-44032.OzT1ob.rst +++ /dev/null @@ -1,2 +0,0 @@ -Move 'fast' locals and other variables from the frame object to a per-thread -datastack. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-15-17-30-57.bpo-44143.7UTS6H.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-15-17-30-57.bpo-44143.7UTS6H.rst deleted file mode 100644 index a4e88e55723e4..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-15-17-30-57.bpo-44143.7UTS6H.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a crash in the parser that manifest when raising tokenizer errors when -an existing exception was present. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-17-20-44-45.bpo-44156.8KSp9l.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-17-20-44-45.bpo-44156.8KSp9l.rst deleted file mode 100644 index 31e63c2b61316..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-17-20-44-45.bpo-44156.8KSp9l.rst +++ /dev/null @@ -1 +0,0 @@ -String caches in ``compile.c`` are now subinterpreter compatible. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-18-11-27-02.bpo-44168.mgB-rt.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-18-11-27-02.bpo-44168.mgB-rt.rst deleted file mode 100644 index ace01e49b508a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-18-11-27-02.bpo-44168.mgB-rt.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix error message in the parser involving keyword arguments with invalid -expressions. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-19-20-33-36.bpo-44180.mQVaAs.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-19-20-33-36.bpo-44180.mQVaAs.rst deleted file mode 100644 index c67eb70b5823a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-19-20-33-36.bpo-44180.mQVaAs.rst +++ /dev/null @@ -1,3 +0,0 @@ -The parser doesn't report generic syntax errors that happen in a position -further away that the one it reached in the first pass. Patch by Pablo -Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-20-12-43-04.bpo-44187.3lk0L1.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-20-12-43-04.bpo-44187.3lk0L1.rst deleted file mode 100644 index 067dedd0f7dda..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-20-12-43-04.bpo-44187.3lk0L1.rst +++ /dev/null @@ -1,3 +0,0 @@ -Implement quickening in the interpreter. This offers no advantages as -yet, but is an enabler of future optimizations. See PEP 659 for full -explanation. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-01-42-45.bpo-44184.9qOptC.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-21-01-42-45.bpo-44184.9qOptC.rst deleted file mode 100644 index 3aba9a58475c6..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-01-42-45.bpo-44184.9qOptC.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a crash at Python exit when a deallocator function removes the last strong -reference to a heap type. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-20-53-49.bpo-43693.-NN3J_.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-21-20-53-49.bpo-43693.-NN3J_.rst deleted file mode 100644 index 83b7ba260e6a2..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-20-53-49.bpo-43693.-NN3J_.rst +++ /dev/null @@ -1,3 +0,0 @@ -``PyCodeObject`` gained ``co_fastlocalnames`` and ``co_fastlocalkinds`` as -the the authoritative source of fast locals info. Marshaled code objects -have changed accordingly. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-21-16-03.bpo-44201.bGaSjt.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-21-21-16-03.bpo-44201.bGaSjt.rst deleted file mode 100644 index 6f61aac7817b2..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-21-21-16-03.bpo-44201.bGaSjt.rst +++ /dev/null @@ -1,3 +0,0 @@ -Avoid side effects of checking for specialized syntax errors in the REPL -that was causing it to ask for extra tokens after a syntax error had been -detected. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-25-18-20-10.bpo-44232.DMcCCf.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-25-18-20-10.bpo-44232.DMcCCf.rst deleted file mode 100644 index fcd124d51442a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-25-18-20-10.bpo-44232.DMcCCf.rst +++ /dev/null @@ -1,4 +0,0 @@ -Fix a regression in :func:`type` when a metaclass raises an exception. The C -function :c:func:`type_new` must properly report the exception when a metaclass -constructor raises an exception and the winner class is not the metaclass. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-26-19-10-47.bpo-43693.1KSG9u.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-26-19-10-47.bpo-43693.1KSG9u.rst deleted file mode 100644 index c5fb29ba462e7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-26-19-10-47.bpo-43693.1KSG9u.rst +++ /dev/null @@ -1,5 +0,0 @@ -A new opcode MAKE_CELL has been added that effectively moves some of -the work done on function entry into the compiler and into the eval -loop. In addition to creating the required cell objects, the new -opcode converts relevant arguments (and other locals) to cell -variables on function entry. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-27-17-34-29.bpo-43667.ND9jP3.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-27-17-34-29.bpo-43667.ND9jP3.rst deleted file mode 100644 index 53130cce7180a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-27-17-34-29.bpo-43667.ND9jP3.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve Unicode support in non-UTF locales on Oracle Solaris. This issue -does not affect other Solaris systems. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-05-30-16-37-47.bpo-43413.vYFPPC.rst b/Misc/NEWS.d/next/Core and Builtins/2021-05-30-16-37-47.bpo-43413.vYFPPC.rst deleted file mode 100644 index 579b57ee2d433..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-05-30-16-37-47.bpo-43413.vYFPPC.rst +++ /dev/null @@ -1,4 +0,0 @@ -Constructors of subclasses of some buitin classes (e.g. :class:`tuple`, -:class:`list`, :class:`frozenset`) no longer accept arbitrary keyword -arguments. Subclass of :class:`set` can now define a ``__new__()`` method -with additional keyword parameters without overriding also ``__init__()``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-03-22-51-50.bpo-44305.66dVDG.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-03-22-51-50.bpo-44305.66dVDG.rst deleted file mode 100644 index eebc26f1cc777..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-03-22-51-50.bpo-44305.66dVDG.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve error message for ``try`` blocks without ``except`` or ``finally`` -blocks. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst deleted file mode 100644 index 89104e8e387ed..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a crash in the :mod:`sqlite3` module that happened when the garbage -collector clears :class:`sqlite.Statement` objects. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-06-00-29-14.bpo-44317.xPPhcZ.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-06-00-29-14.bpo-44317.xPPhcZ.rst deleted file mode 100644 index 8ac32adf8b553..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-06-00-29-14.bpo-44317.xPPhcZ.rst +++ /dev/null @@ -1 +0,0 @@ -Improve tokenizer error with improved locations. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-07-15-13-44.bpo-43693.c_zDeY.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-07-15-13-44.bpo-43693.c_zDeY.rst deleted file mode 100644 index c3db81072254b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-07-15-13-44.bpo-43693.c_zDeY.rst +++ /dev/null @@ -1,4 +0,0 @@ -Computation of the offsets of cell variables is done in the compiler instead -of at runtime. This reduces the overhead of handling cell and free -variables, especially in the case where a variable is both an argument and -cell variable. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-01-13-47.bpo-44335.GQTTkl.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-08-01-13-47.bpo-44335.GQTTkl.rst deleted file mode 100644 index b57904e5da607..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-01-13-47.bpo-44335.GQTTkl.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a regression when identifying incorrect characters in syntax errors. -Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-10-22-46.bpo-44337.RTjmIt.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-08-10-22-46.bpo-44337.RTjmIt.rst deleted file mode 100644 index 2df082a078e30..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-10-22-46.bpo-44337.RTjmIt.rst +++ /dev/null @@ -1,11 +0,0 @@ -Initial implementation of adaptive specialization of LOAD_ATTR - -Four specialized forms of LOAD_ATTR are added: - -* LOAD_ATTR_SLOT - -* LOAD_ATTR_SPLIT_KEYS - -* LOAD_ATTR_WITH_HINT - -* LOAD_ATTR_MODULE diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-22-49-06.bpo-44349.xgEgeA.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-08-22-49-06.bpo-44349.xgEgeA.rst deleted file mode 100644 index b386a8ed2c846..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-08-22-49-06.bpo-44349.xgEgeA.rst +++ /dev/null @@ -1 +0,0 @@ -Fix an edge case when displaying text from files with encoding in syntax errors. Patch by Pablo Galindo. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-09-22-56-59.bpo-44368.vgT0Cx.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-09-22-56-59.bpo-44368.vgT0Cx.rst deleted file mode 100644 index e0d19134510ee..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-09-22-56-59.bpo-44368.vgT0Cx.rst +++ /dev/null @@ -1 +0,0 @@ -Improve syntax errors for invalid "as" targets. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-10-10-06-18.bpo-44338.c4Myr4.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-10-10-06-18.bpo-44338.c4Myr4.rst deleted file mode 100644 index beaa3e56334ba..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-10-10-06-18.bpo-44338.c4Myr4.rst +++ /dev/null @@ -1,7 +0,0 @@ -Implement adaptive specialization for LOAD_GLOBAL - -Two specialized forms of LOAD_GLOBAL are added: - -* LOAD_GLOBAL_MODULE - -* LOAD_GLOBAL_BUILTIN diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-10-16-10-39.bpo-44313.34RjI8.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-10-16-10-39.bpo-44313.34RjI8.rst deleted file mode 100644 index e48d4a471f802..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-10-16-10-39.bpo-44313.34RjI8.rst +++ /dev/null @@ -1,4 +0,0 @@ -Directly imported objects and modules (through import and from import -statements) don't generate ``LOAD_METHOD``/``CALL_METHOD`` for directly -accessed objects on their namespace. They now use the regular -``LOAD_ATTR``/``CALL_FUNCTION``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-11-17-37-15.bpo-44376.zhM1UW.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-11-17-37-15.bpo-44376.zhM1UW.rst deleted file mode 100644 index f854d56b3c841..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-11-17-37-15.bpo-44376.zhM1UW.rst +++ /dev/null @@ -1 +0,0 @@ -Exact integer exponentiation (like ``i**2`` or ``pow(i, 2)``) with a small exponent is much faster, due to reducing overhead in such cases. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-11-18-17-42.bpo-44396.Z9EKim.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-11-18-17-42.bpo-44396.Z9EKim.rst deleted file mode 100644 index be72a7111dc8a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-11-18-17-42.bpo-44396.Z9EKim.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a possible crash in the tokenizer when raising syntax errors for -unclosed strings. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-13-23-12-18.bpo-44409.eW4LS-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-13-23-12-18.bpo-44409.eW4LS-.rst deleted file mode 100644 index 0f204ed812b27..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-13-23-12-18.bpo-44409.eW4LS-.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix error location information for tokenizer errors raised on initialization -of the tokenizer. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-18-22-08-25.bpo-44456.L0Rhko.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-18-22-08-25.bpo-44456.L0Rhko.rst deleted file mode 100644 index 86a8c03ce561e..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-18-22-08-25.bpo-44456.L0Rhko.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve the syntax error when mixing positional and keyword patterns. Patch -by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-19-12-41-13.bpo-44297.F53vHj.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-19-12-41-13.bpo-44297.F53vHj.rst deleted file mode 100644 index bdcb5b2db39e0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-19-12-41-13.bpo-44297.F53vHj.rst +++ /dev/null @@ -1,3 +0,0 @@ -Make sure that the line number is set when entering a comprehension scope. -Ensures that backtraces inclusing generator expressions show the correct -line number. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-20-10-53-21.bpo-12022.SW240M.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-20-10-53-21.bpo-12022.SW240M.rst deleted file mode 100644 index 98c42283169d8..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-20-10-53-21.bpo-12022.SW240M.rst +++ /dev/null @@ -1,4 +0,0 @@ -A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in -:keyword:`with` and :keyword:`async with` statements for objects which do -not support the :term:`context manager` or :term:`asynchronous context -manager` protocols correspondingly. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-21-11-19-54.bpo-44472.Vvm1yn.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-21-11-19-54.bpo-44472.Vvm1yn.rst deleted file mode 100644 index 34fa2a9e8615f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-21-11-19-54.bpo-44472.Vvm1yn.rst +++ /dev/null @@ -1 +0,0 @@ -Fix ltrace functionality when exceptions are raised. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-22-10-55-23.bpo-44486.wct-9X.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-22-10-55-23.bpo-44486.wct-9X.rst deleted file mode 100644 index cc419602541b7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-22-10-55-23.bpo-44486.wct-9X.rst +++ /dev/null @@ -1,2 +0,0 @@ -Modules will always have a dictionary, even when created by -``types.ModuleType.__new__()`` diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-22-19-08-19.bpo-44483.eq2f7T.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-22-19-08-19.bpo-44483.eq2f7T.rst deleted file mode 100644 index ea54e79acfd9d..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-22-19-08-19.bpo-44483.eq2f7T.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a crash in ``types.Union`` objects when creating a union of an object -with bad ``__module__`` field. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-29-11-49-29.bpo-44523.67-ZIP.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-29-11-49-29.bpo-44523.67-ZIP.rst deleted file mode 100644 index aa51a7478583b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-06-29-11-49-29.bpo-44523.67-ZIP.rst +++ /dev/null @@ -1,3 +0,0 @@ -Remove the pass-through for :func:`hash` of :class:`weakref.proxy` objects -to prevent unintended consequences when the original referred object -dies while the proxy is part of a hashable object. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-01-11-59-34.bpo-44490.xY80VR.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-01-11-59-34.bpo-44490.xY80VR.rst deleted file mode 100644 index 4912bca837bb1..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-01-11-59-34.bpo-44490.xY80VR.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ``__parameters__`` attribute and ``__getitem__`` -operator to ``types.Union``. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-02-22-54-41.bpo-44553.l9YqGg.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-02-22-54-41.bpo-44553.l9YqGg.rst deleted file mode 100644 index e97df02994806..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-02-22-54-41.bpo-44553.l9YqGg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Implement GC methods for ``types.Union`` to break reference cycles and -prevent memory leaks. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-03-00-20-39.bpo-43908.YHuV_s.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-03-00-20-39.bpo-43908.YHuV_s.rst deleted file mode 100644 index 6113d0f912a5a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-03-00-20-39.bpo-43908.YHuV_s.rst +++ /dev/null @@ -1,3 +0,0 @@ -Heap types with the :const:`Py_TPFLAGS_IMMUTABLETYPE` flag can now inherit the -:pep:`590` vectorcall protocol. Previously, this was only possible for -:ref:`static types `. Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-04-17-41-47.bpo-41486.DiM24a.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-04-17-41-47.bpo-41486.DiM24a.rst deleted file mode 100644 index 6a373f67f62f9..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-04-17-41-47.bpo-41486.DiM24a.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix a memory consumption and copying performance regression in earlier 3.10 -beta releases if someone used an output buffer larger than 4GiB with -zlib.decompress on input data that expands that large. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-04-23-38-51.bpo-44562.QdeRQo.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-04-23-38-51.bpo-44562.QdeRQo.rst deleted file mode 100644 index 2fc65bcfdeef4..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-04-23-38-51.bpo-44562.QdeRQo.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove uses of :c:func:`PyObject_GC_Del` in error path when initializing -:class:`types.GenericAlias`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst deleted file mode 100644 index dde5399626b7e..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst +++ /dev/null @@ -1,6 +0,0 @@ -Code objects can now provide the column information for instructions when -available. This is levaraged during traceback printing to show the -expressions responsible for errors. - -Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar as part of -:pep:`657`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-06-22-22-15.bpo-44490.BJxPbZ.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-06-22-22-15.bpo-44490.BJxPbZ.rst deleted file mode 100644 index d49181cd82c81..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-06-22-22-15.bpo-44490.BJxPbZ.rst +++ /dev/null @@ -1,3 +0,0 @@ -:mod:`typing` now searches for type parameters in ``types.Union`` objects. -``get_type_hints`` will also properly resolve annotations with nested -``types.Union`` objects. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-07-16-05-35.bpo-43895.JFjR0-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-07-16-05-35.bpo-43895.JFjR0-.rst deleted file mode 100644 index 49deb48fa4358..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-07-16-05-35.bpo-43895.JFjR0-.rst +++ /dev/null @@ -1,4 +0,0 @@ -An obsolete internal cache of shared object file handles added in 1995 that -attempted, but did not guarantee, that a .so would not be dlopen'ed twice to -work around flaws in mid-1990s posix-ish operating systems has been removed -from dynload_shlib.c. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-08-12-18-56.bpo-44584.qKnSqV.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-08-12-18-56.bpo-44584.qKnSqV.rst deleted file mode 100644 index 4afb33b047d4b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-08-12-18-56.bpo-44584.qKnSqV.rst +++ /dev/null @@ -1,3 +0,0 @@ -The threading debug (:envvar:`PYTHONTHREADDEBUG` environment variable) is -deprecated in Python 3.10 and will be removed in Python 3.12. This feature -requires a debug build of Python. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-09-12-08-17.bpo-44590.a2ntVX.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-09-12-08-17.bpo-44590.a2ntVX.rst deleted file mode 100644 index ed4d969a11ab4..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-09-12-08-17.bpo-44590.a2ntVX.rst +++ /dev/null @@ -1,5 +0,0 @@ -All necessary data for executing a Python function (local variables, stack, -etc) is now kept in a per-thread stack. Frame objects are lazily allocated -on demand. This increases performance by about 7% on the standard benchmark -suite. Introspection and debugging are unaffected as frame objects are -always available when needed. Patch by Mark Shannon. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-12-04-06-57.bpo-41972.nDX5k_.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-12-04-06-57.bpo-41972.nDX5k_.rst deleted file mode 100644 index 3daffb9c0e1df..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-12-04-06-57.bpo-41972.nDX5k_.rst +++ /dev/null @@ -1 +0,0 @@ -Tuned the string-searching algorithm of fastsearch.h to have a shorter inner loop for most cases. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-17-47-32.bpo-42073.9wopiC.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-13-17-47-32.bpo-42073.9wopiC.rst deleted file mode 100644 index 988fe67b99dc9..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-17-47-32.bpo-42073.9wopiC.rst +++ /dev/null @@ -1,2 +0,0 @@ -The ``@classmethod`` decorator can now wrap other classmethod-like -descriptors. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-20-22-12.bpo-44606.S3Bv2w.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-13-20-22-12.bpo-44606.S3Bv2w.rst deleted file mode 100644 index 3c7ef3b21fe09..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-20-22-12.bpo-44606.S3Bv2w.rst +++ /dev/null @@ -1 +0,0 @@ -Fix ``__instancecheck__`` and ``__subclasscheck__`` for the union type. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-23-19-41.bpo-44589.59OH8T.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-13-23-19-41.bpo-44589.59OH8T.rst deleted file mode 100644 index 23b2472b00e40..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-13-23-19-41.bpo-44589.59OH8T.rst +++ /dev/null @@ -1,2 +0,0 @@ -Mapping patterns in ``match`` statements with two or more equal literal -keys will now raise a :exc:`SyntaxError` at compile-time. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-14-10-31-10.bpo-26280.cgpM4B.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-14-10-31-10.bpo-26280.cgpM4B.rst deleted file mode 100644 index cb561e79c78ce..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-14-10-31-10.bpo-26280.cgpM4B.rst +++ /dev/null @@ -1,9 +0,0 @@ -Implement adaptive specialization for BINARY_SUBSCR - - Three specialized forms of BINARY_SUBSCR are added: - - * BINARY_SUBSCR_LIST_INT - - * BINARY_SUBSCR_TUPLE_INT - - * BINARY_SUBSCR_DICT \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-14-13-54-07.bpo-44635.7ZMAdB.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-14-13-54-07.bpo-44635.7ZMAdB.rst deleted file mode 100644 index ea00554aeeda6..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-14-13-54-07.bpo-44635.7ZMAdB.rst +++ /dev/null @@ -1 +0,0 @@ -Convert ``None`` to ``type(None)`` in the union type constructor. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-01-01-11.bpo-44611.LcfHN-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-16-01-01-11.bpo-44611.LcfHN-.rst deleted file mode 100644 index 1cc8b12738239..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-01-01-11.bpo-44611.LcfHN-.rst +++ /dev/null @@ -1,2 +0,0 @@ -On Windows, :func:`os.urandom`: uses BCryptGenRandom API instead of CryptGenRandom API -which is deprecated from Microsoft Windows API. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-36-12.bpo-44636.ZWebi8.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-36-12.bpo-44636.ZWebi8.rst deleted file mode 100644 index 1a053ae69e97e..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-36-12.bpo-44636.ZWebi8.rst +++ /dev/null @@ -1 +0,0 @@ -Collapse union of equal types. E.g. the result of ``int | int`` is now ``int``. Fix comparison of the union type with non-hashable objects. E.g. ``int | str == {}`` no longer raises a TypeError. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-59-13.bpo-44646.Yb6s05.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-59-13.bpo-44646.Yb6s05.rst deleted file mode 100644 index 0e28eac54fe98..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-09-59-13.bpo-44646.Yb6s05.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix the hash of the union type: it no longer depends on the order of -arguments. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-20-25-37.bpo-44655.I3wRjL.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-16-20-25-37.bpo-44655.I3wRjL.rst deleted file mode 100644 index 4ea4a6d085979..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-20-25-37.bpo-44655.I3wRjL.rst +++ /dev/null @@ -1,2 +0,0 @@ -Don't include a missing attribute with the same name as the failing one when -offering suggestions for missing attributes. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-21-35-14.bpo-44655.95I7M6.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-16-21-35-14.bpo-44655.95I7M6.rst deleted file mode 100644 index 17733b3619a8f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-16-21-35-14.bpo-44655.95I7M6.rst +++ /dev/null @@ -1,2 +0,0 @@ -Include the name of the type in unset __slots__ attribute errors. Patch by -Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-13-41-58.bpo-44662.q22kWR.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-17-13-41-58.bpo-44662.q22kWR.rst deleted file mode 100644 index c165774a4cacf..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-13-41-58.bpo-44662.q22kWR.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add ``__module__`` to ``types.Union``. This also fixes -``types.Union`` issues with ``typing.Annotated``. Patch provided by -Yurii Karabas. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-14-20-59.bpo-44661.BQbXiH.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-17-14-20-59.bpo-44661.BQbXiH.rst deleted file mode 100644 index bafa98e5826cd..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-14-20-59.bpo-44661.BQbXiH.rst +++ /dev/null @@ -1,2 +0,0 @@ -Update ``property_descr_set`` to use vectorcall if possible. Patch by Dong-hee -Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-21-04-04.bpo-44633.5-zKeI.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-17-21-04-04.bpo-44633.5-zKeI.rst deleted file mode 100644 index 507a68b65d4c3..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-17-21-04-04.bpo-44633.5-zKeI.rst +++ /dev/null @@ -1,2 +0,0 @@ -Parameter substitution of the union type with wrong types now raises -``TypeError`` instead of returning ``NotImplemented``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst deleted file mode 100644 index 4a1815e041dcc..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-19-19-53-46.bpo-44676.WgIMvh.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ability to serialise ``types.Union`` objects. Patch provided by Yurii -Karabas. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-19-20-49-06.bpo-44653.WcqGyI.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-19-20-49-06.bpo-44653.WcqGyI.rst deleted file mode 100644 index 8254d9bbad4a6..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-19-20-49-06.bpo-44653.WcqGyI.rst +++ /dev/null @@ -1 +0,0 @@ -Support :mod:`typing` types in parameter substitution in the union type. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-21-15-26-56.bpo-44698.DA4_0o.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-21-15-26-56.bpo-44698.DA4_0o.rst deleted file mode 100644 index ed389630c8ba1..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-21-15-26-56.bpo-44698.DA4_0o.rst +++ /dev/null @@ -1 +0,0 @@ -Fix undefined behaviour in complex object exponentiation. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-23-01-52-13.bpo-44717.-vVmAh.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-23-01-52-13.bpo-44717.-vVmAh.rst deleted file mode 100644 index 7a4d77f7d6520..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-23-01-52-13.bpo-44717.-vVmAh.rst +++ /dev/null @@ -1 +0,0 @@ -Improve AttributeError on circular imports of submodules. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-23-15-17-01.bpo-44725.qcuKaa.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-23-15-17-01.bpo-44725.qcuKaa.rst deleted file mode 100644 index 995cf14800143..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-23-15-17-01.bpo-44725.qcuKaa.rst +++ /dev/null @@ -1 +0,0 @@ -Expose specialization stats in python via :func:`_opcode.get_specialization_stats`. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-26-15-27-03.bpo-44732.IxObt3.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-26-15-27-03.bpo-44732.IxObt3.rst deleted file mode 100644 index 5094688a29431..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-26-15-27-03.bpo-44732.IxObt3.rst +++ /dev/null @@ -1 +0,0 @@ -Rename ``types.Union`` to ``types.UnionType``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-27-11-14-22.bpo-34013.SjLFe-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-27-11-14-22.bpo-34013.SjLFe-.rst deleted file mode 100644 index c0d3dd99f9827..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-27-11-14-22.bpo-34013.SjLFe-.rst +++ /dev/null @@ -1,3 +0,0 @@ -Generalize the invalid legacy statement custom error message (like the one -generated when "print" is called without parentheses) to include more -generic expressions. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-31-12-12-57.bpo-44792.mOReTW.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-31-12-12-57.bpo-44792.mOReTW.rst deleted file mode 100644 index 2e9000d09ee44..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-07-31-12-12-57.bpo-44792.mOReTW.rst +++ /dev/null @@ -1 +0,0 @@ -Improve syntax errors for if expressions. Patch by Miguel Brito diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-04-11-37-38.bpo-44821.67YHGI.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-04-11-37-38.bpo-44821.67YHGI.rst deleted file mode 100644 index 1e254a62773bb..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-04-11-37-38.bpo-44821.67YHGI.rst +++ /dev/null @@ -1,2 +0,0 @@ -Create instance dictionaries (__dict__) eagerly, to improve regularity of -object layout and assist specialization. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-42-03.bpo-44838.r_Lkj_.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-42-03.bpo-44838.r_Lkj_.rst deleted file mode 100644 index a542a5d70933a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-42-03.bpo-44838.r_Lkj_.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a bug that was causing the parser to raise an incorrect custom -:exc:`SyntaxError` for invalid 'if' expressions. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-49-55.bpo-44826.zQsyK5.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-49-55.bpo-44826.zQsyK5.rst deleted file mode 100644 index ccdb5e0c606c0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-05-17-49-55.bpo-44826.zQsyK5.rst +++ /dev/null @@ -1,9 +0,0 @@ -Initial implementation of adaptive specialization of STORE_ATTR - -Three specialized forms of STORE_ATTR are added: - -* STORE_ATTR_SLOT - -* STORE_ATTR_SPLIT_KEYS - -* STORE_ATTR_WITH_HINT diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-07-01-26-12.bpo-44856.9rk3li.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-07-01-26-12.bpo-44856.9rk3li.rst deleted file mode 100644 index 1111d01b726fa..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-07-01-26-12.bpo-44856.9rk3li.rst +++ /dev/null @@ -1 +0,0 @@ -Fix reference leaks in the error paths of ``update_bases()`` and ``__build_class__``. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-07-21-39-19.bpo-25782.B22lMx.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-07-21-39-19.bpo-25782.B22lMx.rst deleted file mode 100644 index 1c52059f76c8f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-07-21-39-19.bpo-25782.B22lMx.rst +++ /dev/null @@ -1 +0,0 @@ -Fix bug where ``PyErr_SetObject`` hangs when the current exception has a cycle in its context chain. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-14-29-52.bpo-33930.--5LQ-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-09-14-29-52.bpo-33930.--5LQ-.rst deleted file mode 100644 index 827dd3f8b6513..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-14-29-52.bpo-33930.--5LQ-.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix segmentation fault with deep recursion when cleaning method objects. -Patch by Augusto Goulart and Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-16-16-03.bpo-44872.OKRlhK.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-09-16-16-03.bpo-44872.OKRlhK.rst deleted file mode 100644 index 9a0d00018b2a7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-16-16-03.bpo-44872.OKRlhK.rst +++ /dev/null @@ -1 +0,0 @@ -Use new trashcan macros (Py_TRASHCAN_BEGIN/END) in frameobject.c instead of the old ones (Py_TRASHCAN_SAFE_BEGIN/END). \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-19-05-20.bpo-44874.oOcfU4.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-09-19-05-20.bpo-44874.oOcfU4.rst deleted file mode 100644 index 1aed5351d7417..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-09-19-05-20.bpo-44874.oOcfU4.rst +++ /dev/null @@ -1 +0,0 @@ -Deprecate the old trashcan macros (``Py_TRASHCAN_SAFE_BEGIN``/``Py_TRASHCAN_SAFE_END``). They should be replaced by the new macros ``Py_TRASHCAN_BEGIN`` and ``Py_TRASHCAN_END``. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-12-03-52.bpo-44878.nEhjLi.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-11-12-03-52.bpo-44878.nEhjLi.rst deleted file mode 100644 index 7998140d4aab0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-12-03-52.bpo-44878.nEhjLi.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove switch statement for interpreter loop when using computed gotos. This -makes sure that we only have one dispatch table in the interpreter. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-14-12-41.bpo-44878.pAbBfc.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-11-14-12-41.bpo-44878.pAbBfc.rst deleted file mode 100644 index 9e07fa8a25df0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-14-12-41.bpo-44878.pAbBfc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove the loop from the bytecode interpreter. All instructions end with a -DISPATCH macro, so the loop is now redundant. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-15-39-57.bpo-44885.i4noUO.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-11-15-39-57.bpo-44885.i4noUO.rst deleted file mode 100644 index c6abd7363af71..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-15-39-57.bpo-44885.i4noUO.rst +++ /dev/null @@ -1,2 +0,0 @@ -Correct the ast locations of f-strings with format specs and repeated -expressions. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-16-46-27.bpo-44890.PwNg8N.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-11-16-46-27.bpo-44890.PwNg8N.rst deleted file mode 100644 index 48a1c8b690e65..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-16-46-27.bpo-44890.PwNg8N.rst +++ /dev/null @@ -1 +0,0 @@ -Specialization stats are always collected in debug builds. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-20-45-02.bpo-44889.2T3nTn.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-11-20-45-02.bpo-44889.2T3nTn.rst deleted file mode 100644 index a50b6851c14b7..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-11-20-45-02.bpo-44889.2T3nTn.rst +++ /dev/null @@ -1,8 +0,0 @@ -Initial implementation of adaptive specialization of ``LOAD_METHOD``. The -following specialized forms were added: - -* ``LOAD_METHOD_CACHED`` - -* ``LOAD_METHOD_MODULE`` - -* ``LOAD_METHOD_CLASS`` diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-12-14-00-57.bpo-44900.w2gpwy.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-12-14-00-57.bpo-44900.w2gpwy.rst deleted file mode 100644 index 8d94d6aed80ea..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-12-14-00-57.bpo-44900.w2gpwy.rst +++ /dev/null @@ -1,7 +0,0 @@ -Add five superinstructions for PEP 659 quickening: - -* LOAD_FAST LOAD_FAST -* STORE_FAST LOAD_FAST -* LOAD_FAST LOAD_CONST -* LOAD_CONST LOAD_FAST -* STORE_FAST STORE_FAST diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-14-20-13-21.bpo-44895.Ic9m90.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-14-20-13-21.bpo-44895.Ic9m90.rst deleted file mode 100644 index d369ac7650597..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-14-20-13-21.bpo-44895.Ic9m90.rst +++ /dev/null @@ -1,2 +0,0 @@ -A debug variable :envvar:`PYTHONDUMPREFSFILE` is added for creating a dump file -which is generated by :option:`--with-trace-refs`. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-15-10-39-06.bpo-44698.lITKNc.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-15-10-39-06.bpo-44698.lITKNc.rst deleted file mode 100644 index f197253e10ec6..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-15-10-39-06.bpo-44698.lITKNc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Restore behaviour of complex exponentiation with integer-valued exponent of -type :class:`float` or :class:`complex`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-16-11-36-02.bpo-44914.6Lgrx3.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-16-11-36-02.bpo-44914.6Lgrx3.rst deleted file mode 100644 index 5e306ffaf5de8..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-16-11-36-02.bpo-44914.6Lgrx3.rst +++ /dev/null @@ -1,5 +0,0 @@ -Class version tags are no longer recycled. - -This means that a version tag serves as a unique identifier for the state of -a class. We rely on this for effective specialization of the LOAD_ATTR and -other instructions. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-16-23-16-17.bpo-44929.qpMEky.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-16-23-16-17.bpo-44929.qpMEky.rst deleted file mode 100644 index e883e031a4afd..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-16-23-16-17.bpo-44929.qpMEky.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix some edge cases of ``enum.Flag`` string representation in the REPL. -Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-18-11-14-38.bpo-44945.CO3s77.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-18-11-14-38.bpo-44945.CO3s77.rst deleted file mode 100644 index 66d53ec523de3..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-18-11-14-38.bpo-44945.CO3s77.rst +++ /dev/null @@ -1,7 +0,0 @@ -Specialize the BINARY_ADD instruction using the PEP 659 machinery. Adds five new instructions: - -* BINARY_ADD_ADAPTIVE -* BINARY_ADD_FLOAT -* BINARY_ADD_INT -* BINARY_ADD_UNICODE -* BINARY_ADD_UNICODE_INPLACE_FAST diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-18-19-09-28.bpo-44947.mcvGdS.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-18-19-09-28.bpo-44947.mcvGdS.rst deleted file mode 100644 index d531ba9faf3de..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-18-19-09-28.bpo-44947.mcvGdS.rst +++ /dev/null @@ -1,2 +0,0 @@ -Refine the syntax error for trailing commas in import statements. Patch by -Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-19-14-43-24.bpo-44954.dLn3lg.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-19-14-43-24.bpo-44954.dLn3lg.rst deleted file mode 100644 index 4cdeb34b8b611..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-19-14-43-24.bpo-44954.dLn3lg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a corner case bug where the result of ``float.fromhex('0x.8p-1074')`` -was rounded the wrong way. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-22-12-28-50.bpo-24234.n3oTdx.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-22-12-28-50.bpo-24234.n3oTdx.rst deleted file mode 100644 index 52397e90fbdfa..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-22-12-28-50.bpo-24234.n3oTdx.rst +++ /dev/null @@ -1,3 +0,0 @@ -Implement the :meth:`__complex__` special method on the :class:`complex` type, -so a complex number ``z`` passes an ``isinstance(z, typing.SupportsComplex)`` -check. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-23-10-36-55.bpo-24234.MGVUQi.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-23-10-36-55.bpo-24234.MGVUQi.rst deleted file mode 100644 index 3f724da08898a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-23-10-36-55.bpo-24234.MGVUQi.rst +++ /dev/null @@ -1,3 +0,0 @@ -Implement the :meth:`__bytes__` special method on the :class:`bytes` type, -so a bytes object ``b`` passes an ``isinstance(b, typing.SupportsBytes)`` -check. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-23-19-55-08.bpo-44962.J00ftt.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-23-19-55-08.bpo-44962.J00ftt.rst deleted file mode 100644 index 6b4b9dfd8bc3c..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-23-19-55-08.bpo-44962.J00ftt.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a race in WeakKeyDictionary, WeakValueDictionary and WeakSet when two threads attempt to commit the last pending removal. This fixes asyncio.create_task and fixes a data loss in asyncio.run where shutdown_asyncgens is not run \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-07-10.bpo-44963.5EET8y.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-07-10.bpo-44963.5EET8y.rst deleted file mode 100644 index 9a54bda118e00..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-07-10.bpo-44963.5EET8y.rst +++ /dev/null @@ -1,2 +0,0 @@ -Implement ``send()`` and ``throw()`` methods for ``anext_awaitable`` -objects. Patch by Pablo Galindo. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-17-32.bpo-45000.XjmyLl.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-17-32.bpo-45000.XjmyLl.rst deleted file mode 100644 index 96c95cc6e0296..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-25-23-17-32.bpo-45000.XjmyLl.rst +++ /dev/null @@ -1,2 +0,0 @@ -A :exc:`SyntaxError` is now raised when trying to delete :const:`__debug__`. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-26-18-44-03.bpo-45018.pu8H9L.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-26-18-44-03.bpo-45018.pu8H9L.rst deleted file mode 100644 index 88ef80630ef9b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-26-18-44-03.bpo-45018.pu8H9L.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed pickling of range iterators that iterated for over ``2**32`` times. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-31-11-09-52.bpo-45012.ueeOcx.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-31-11-09-52.bpo-45012.ueeOcx.rst deleted file mode 100644 index 91cb3a9e69cc5..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-31-11-09-52.bpo-45012.ueeOcx.rst +++ /dev/null @@ -1,2 +0,0 @@ -In :mod:`posix`, release GIL during ``stat()``, ``lstat()``, and -``fstatat()`` syscalls made by :func:`os.DirEntry.stat`. Patch by Stanis?aw Skonieczny. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-08-31-17-44-51.bpo-45020.ZPI_3L.rst b/Misc/NEWS.d/next/Core and Builtins/2021-08-31-17-44-51.bpo-45020.ZPI_3L.rst deleted file mode 100644 index f6dffa0831c54..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-08-31-17-44-51.bpo-45020.ZPI_3L.rst +++ /dev/null @@ -1,3 +0,0 @@ -Add a new command line option, "-X frozen_modules=[on|off]" to opt out -of (or into) using optional frozen modules. This defaults to "on" (or -"off" if it's a debug build). diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-16-55-43.bpo-45056.7AK2d9.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-01-16-55-43.bpo-45056.7AK2d9.rst deleted file mode 100644 index 6c790f5c50c48..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-16-55-43.bpo-45056.7AK2d9.rst +++ /dev/null @@ -1 +0,0 @@ -Compiler now removes trailing unused constants from co_consts. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-19-21-48.bpo-34561.uMAVA-.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-01-19-21-48.bpo-34561.uMAVA-.rst deleted file mode 100644 index 7c48cb39df1c5..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-19-21-48.bpo-34561.uMAVA-.rst +++ /dev/null @@ -1 +0,0 @@ -List sorting now uses the merge-ordering strategy from Munro and Wild's ``powersort()``. Unlike the former strategy, this is provably near-optimal in the entropy of the distribution of run lengths. Most uses of ``list.sort()`` probably won't see a significant time difference, but may see significant improvements in cases where the former strategy was exceptionally poor. However, as these are all fast linear-time approximations to a problem that's inherently at best quadratic-time to solve truly optimally, it's also possible to contrive cases where the former strategy did better. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-23-55-49.bpo-45083.cLi9G3.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-01-23-55-49.bpo-45083.cLi9G3.rst deleted file mode 100644 index 7bfd87b942059..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-01-23-55-49.bpo-45083.cLi9G3.rst +++ /dev/null @@ -1,3 +0,0 @@ -When the interpreter renders an exception, its name now has a complete qualname. Previously only the class name was concatenated to the module name, which sometimes resulted in an incorrect full name being displayed. - -(This issue impacted only the C code exception rendering, the :mod:`traceback` module was using qualname already). \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-02-01-28-01.bpo-37330.QDjM_l.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-02-01-28-01.bpo-37330.QDjM_l.rst deleted file mode 100644 index 3f09449de70d0..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-02-01-28-01.bpo-37330.QDjM_l.rst +++ /dev/null @@ -1,4 +0,0 @@ -:func:`open`, :func:`io.open`, :func:`codecs.open` and -:class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") -in the file mode. This flag was deprecated since Python 3.3. Patch by Victor -Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-03-12-35-17.bpo-41031.yPSJEs.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-03-12-35-17.bpo-41031.yPSJEs.rst deleted file mode 100644 index 5dcfaa0046c65..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-03-12-35-17.bpo-41031.yPSJEs.rst +++ /dev/null @@ -1 +0,0 @@ -Match C and Python code formatting of unprintable exceptions and exceptions in the :mod:`__main__` module. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-03-16-18-10.bpo-1514420.2Lumpj.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-03-16-18-10.bpo-1514420.2Lumpj.rst deleted file mode 100644 index fdd5cd70c5c2f..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-03-16-18-10.bpo-1514420.2Lumpj.rst +++ /dev/null @@ -1 +0,0 @@ -Interpreter no longer attempts to open files with names in angle brackets (like "" or "") when formatting an exception. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-06-21-52-45.bpo-45123.8Eh9iI.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-06-21-52-45.bpo-45123.8Eh9iI.rst deleted file mode 100644 index 6cc7303766f25..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-06-21-52-45.bpo-45123.8Eh9iI.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix PyAiter_Check to only check for the __anext__ presence (not for -__aiter__). Rename PyAiter_Check to PyAIter_Check, PyObject_GetAiter -> -PyObject_GetAIter. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-07-00-21-04.bpo-44348.f8w_Td.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-07-00-21-04.bpo-44348.f8w_Td.rst deleted file mode 100644 index c222a07725b8a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-07-00-21-04.bpo-44348.f8w_Td.rst +++ /dev/null @@ -1,8 +0,0 @@ -The deallocator function of the :exc:`BaseException` type now uses the -trashcan mecanism to prevent stack overflow. For example, when a -:exc:`RecursionError` instance is raised, it can be linked to another -RecursionError through the ``__context__`` attribute or the -``__traceback__`` attribute, and then a chain of exceptions is created. When -the chain is destroyed, nested deallocator function calls can crash with a -stack overflow if the chain is too long compared to the available stack -memory. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-07-17-10-16.bpo-45121.iG-Hsf.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-07-17-10-16.bpo-45121.iG-Hsf.rst deleted file mode 100644 index 19eb331412516..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-07-17-10-16.bpo-45121.iG-Hsf.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix issue where ``Protocol.__init__`` raises ``RecursionError`` when it's -called directly or via ``super()``. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst deleted file mode 100644 index 02e11ae94e430..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst +++ /dev/null @@ -1 +0,0 @@ -Added fallback to extension modules with '.sl' suffix on HP-UX \ No newline at end of file diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-09-10-32-33.bpo-44219.WiYyjz.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-09-10-32-33.bpo-44219.WiYyjz.rst deleted file mode 100644 index 2abd81673663b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-09-10-32-33.bpo-44219.WiYyjz.rst +++ /dev/null @@ -1,5 +0,0 @@ -Release the GIL while performing ``isatty`` system calls on arbitrary file -descriptors. In particular, this affects :func:`os.isatty`, -:func:`os.device_encoding` and :class:`io.TextIOWrapper`. By extension, -:func:`io.open` in text mode is also affected. This change solves -a deadlock in :func:`os.isatty`. Patch by Vincent Michel in :issue:`44219`. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-09-15-05-17.bpo-45155.JRw9TG.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-09-15-05-17.bpo-45155.JRw9TG.rst deleted file mode 100644 index eab023bf89471..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-09-15-05-17.bpo-45155.JRw9TG.rst +++ /dev/null @@ -1,3 +0,0 @@ -:meth:`int.to_bytes` and :meth:`int.from_bytes` now take a default value of -``"big"`` for the ``byteorder`` argument. :meth:`int.to_bytes` also takes a -default value of ``1`` for the ``length`` argument. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-14-09-23-59.bpo-45167.CPSSoV.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-14-09-23-59.bpo-45167.CPSSoV.rst deleted file mode 100644 index 47755ae59be2b..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-14-09-23-59.bpo-45167.CPSSoV.rst +++ /dev/null @@ -1 +0,0 @@ -Fix deepcopying of :class:`types.GenericAlias` objects. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-14-10-02-12.bpo-45190.ZFRgSj.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-14-10-02-12.bpo-45190.ZFRgSj.rst deleted file mode 100644 index c6a4c554aff5a..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-14-10-02-12.bpo-45190.ZFRgSj.rst +++ /dev/null @@ -1 +0,0 @@ -Update Unicode databases to Unicode 14.0.0. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-20-10-02-12.bpo-24076.ZFgFSj.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-20-10-02-12.bpo-24076.ZFgFSj.rst deleted file mode 100644 index b680884ff8b1e..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-20-10-02-12.bpo-24076.ZFgFSj.rst +++ /dev/null @@ -1 +0,0 @@ -sum() was further optimised for summing up single digit integers. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-21-22-27-25.bpo-45061.5IOUf0.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-21-22-27-25.bpo-45061.5IOUf0.rst deleted file mode 100644 index caeb36ba52646..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-09-21-22-27-25.bpo-45061.5IOUf0.rst +++ /dev/null @@ -1,5 +0,0 @@ -Add a deallocator to the bool type to detect refcount bugs in C extensions -which call Py_DECREF(Py_True) or Py_DECREF(Py_False) by mistake. Detect also -refcount bugs when the empty tuple singleton or the Unicode empty string -singleton is destroyed by mistake. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst deleted file mode 100644 index 1809b42b94438..0000000000000 --- a/Misc/NEWS.d/next/Core and Builtins/2021-10-04-16-11-50.bpo-43760.R9QoUv.rst +++ /dev/null @@ -1,3 +0,0 @@ -The number of hardware branches per instruction dispatch is reduced from two -to one by adding a special instruction for tracing. Patch by Mark Shannon. - diff --git a/Misc/NEWS.d/next/Documentation/2018-05-19-15-59-29.bpo-33479.4cLlxo.rst b/Misc/NEWS.d/next/Documentation/2018-05-19-15-59-29.bpo-33479.4cLlxo.rst deleted file mode 100644 index db4973d392395..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2018-05-19-15-59-29.bpo-33479.4cLlxo.rst +++ /dev/null @@ -1,4 +0,0 @@ -Remove the unqualified claim that tkinter is threadsafe. It has not been -true for several years and likely never was. An explanation of what is true -may be added later, after more discussion, and possibly after patching -_tkinter.c, diff --git a/Misc/NEWS.d/next/Documentation/2020-01-30-05-18-48.bpo-39498.Nu3sFL.rst b/Misc/NEWS.d/next/Documentation/2020-01-30-05-18-48.bpo-39498.Nu3sFL.rst deleted file mode 100644 index a3e899a80a0fc..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-01-30-05-18-48.bpo-39498.Nu3sFL.rst +++ /dev/null @@ -1 +0,0 @@ -Add a "Security Considerations" index which links to standard library modules that have explicitly documented security considerations. diff --git a/Misc/NEWS.d/next/Documentation/2020-03-21-01-19-28.bpo-21760.CqofIc.rst b/Misc/NEWS.d/next/Documentation/2020-03-21-01-19-28.bpo-21760.CqofIc.rst deleted file mode 100644 index 119ef3d4c4378..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-03-21-01-19-28.bpo-21760.CqofIc.rst +++ /dev/null @@ -1,2 +0,0 @@ -The description for __file__ fixed. -Patch by Furkan Onder \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2020-08-21-22-59-37.bpo-41576.7a6CQR.rst b/Misc/NEWS.d/next/Documentation/2020-08-21-22-59-37.bpo-41576.7a6CQR.rst deleted file mode 100644 index f74ef62ca47ab..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-08-21-22-59-37.bpo-41576.7a6CQR.rst +++ /dev/null @@ -1 +0,0 @@ -document BaseException in favor of bare except \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2020-08-24-13-35-04.bpo-41621.nqaw9G.rst b/Misc/NEWS.d/next/Documentation/2020-08-24-13-35-04.bpo-41621.nqaw9G.rst deleted file mode 100644 index bd193d9163073..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-08-24-13-35-04.bpo-41621.nqaw9G.rst +++ /dev/null @@ -1 +0,0 @@ -Document that :class:`collections.defaultdict` parameter ``default_factory`` defaults to None and is positional-only. diff --git a/Misc/NEWS.d/next/Documentation/2020-09-03-13-37-19.bpo-41706._zXWOR.rst b/Misc/NEWS.d/next/Documentation/2020-09-03-13-37-19.bpo-41706._zXWOR.rst deleted file mode 100644 index 6406494ec03c7..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2020-09-03-13-37-19.bpo-41706._zXWOR.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix docs about how methods like ``__add__`` are invoked when evaluating -operator expressions. diff --git a/Misc/NEWS.d/next/Documentation/2021-05-03-22-08-08.bpo-44025.gcB7iP.rst b/Misc/NEWS.d/next/Documentation/2021-05-03-22-08-08.bpo-44025.gcB7iP.rst deleted file mode 100644 index 1432236a32f63..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-03-22-08-08.bpo-44025.gcB7iP.rst +++ /dev/null @@ -1 +0,0 @@ -Clarify when '_' in match statements is a keyword, and when not. diff --git a/Misc/NEWS.d/next/Documentation/2021-05-07-12-27-09.bpo-43558.UGhA8R.rst b/Misc/NEWS.d/next/Documentation/2021-05-07-12-27-09.bpo-43558.UGhA8R.rst deleted file mode 100644 index b0ecb171ef731..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-07-12-27-09.bpo-43558.UGhA8R.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add the remark to :mod:`dataclasses` documentation that the :meth:`__init__` of any base class -has to be called in :meth:`__post_init__`, along with a code example. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2021-05-08-09-48-05.bpo-44072.fb2x5I.rst b/Misc/NEWS.d/next/Documentation/2021-05-08-09-48-05.bpo-44072.fb2x5I.rst deleted file mode 100644 index a5b0c95d85e66..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-08-09-48-05.bpo-44072.fb2x5I.rst +++ /dev/null @@ -1,2 +0,0 @@ -Correct where in the numeric ABC hierarchy ``**`` support is added, i.e., in -numbers.Complex, not numbers.Integral. diff --git a/Misc/NEWS.d/next/Documentation/2021-05-17-20-03-47.bpo-41963.eUz9_o.rst b/Misc/NEWS.d/next/Documentation/2021-05-17-20-03-47.bpo-41963.eUz9_o.rst deleted file mode 100644 index b9fe722fa3a79..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-17-20-03-47.bpo-41963.eUz9_o.rst +++ /dev/null @@ -1 +0,0 @@ -Document that ``ConfigParser`` strips off comments when reading configuration files. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2021-05-23-09-11-28.bpo-44195.1bqkOs.rst b/Misc/NEWS.d/next/Documentation/2021-05-23-09-11-28.bpo-44195.1bqkOs.rst deleted file mode 100644 index 5f165f166a377..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-23-09-11-28.bpo-44195.1bqkOs.rst +++ /dev/null @@ -1,2 +0,0 @@ -Corrected references to ``TraversableResources`` in docs. There is no -``TraversableReader``. diff --git a/Misc/NEWS.d/next/Documentation/2021-05-26-11-16-33.bpo-42392.oxRx6E.rst b/Misc/NEWS.d/next/Documentation/2021-05-26-11-16-33.bpo-42392.oxRx6E.rst deleted file mode 100644 index 5c840de6f68ef..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-05-26-11-16-33.bpo-42392.oxRx6E.rst +++ /dev/null @@ -1,2 +0,0 @@ -Document the deprecation and removal of the ``loop`` parameter for many -functions and classes in :mod:`asyncio`. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-06-14-12-00.bpo-44322.K0PHfE.rst b/Misc/NEWS.d/next/Documentation/2021-06-06-14-12-00.bpo-44322.K0PHfE.rst deleted file mode 100644 index 48dd7e6d97662..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-06-14-12-00.bpo-44322.K0PHfE.rst +++ /dev/null @@ -1,2 +0,0 @@ -Document that SyntaxError args have a details tuple and that details are -adjusted for errors in f-string field replacement expressions. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-14-09-20-37.bpo-38291.VMYa_Q.rst b/Misc/NEWS.d/next/Documentation/2021-06-14-09-20-37.bpo-38291.VMYa_Q.rst deleted file mode 100644 index 23ce35eb176d9..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-14-09-20-37.bpo-38291.VMYa_Q.rst +++ /dev/null @@ -1,2 +0,0 @@ -Mark ``typing.io`` and ``typing.re`` as deprecated since Python 3.8 in the -documentation. They were never properly supported by type checkers. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-16-18-09-49.bpo-44392.6RF1Sc.rst b/Misc/NEWS.d/next/Documentation/2021-06-16-18-09-49.bpo-44392.6RF1Sc.rst deleted file mode 100644 index ac197f22929d1..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-16-18-09-49.bpo-44392.6RF1Sc.rst +++ /dev/null @@ -1,2 +0,0 @@ -Added a new section in the C API documentation for types used in type -hinting. Documented ``Py_GenericAlias`` and ``Py_GenericAliasType``. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-18-06-44-45.bpo-44453.3PIkj2.rst b/Misc/NEWS.d/next/Documentation/2021-06-18-06-44-45.bpo-44453.3PIkj2.rst deleted file mode 100644 index fd72cf525c32f..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-18-06-44-45.bpo-44453.3PIkj2.rst +++ /dev/null @@ -1 +0,0 @@ -Fix documentation for the return type of :func:`sysconfig.get_path`. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-18-18-04-53.bpo-27752.NEByNk.rst b/Misc/NEWS.d/next/Documentation/2021-06-18-18-04-53.bpo-27752.NEByNk.rst deleted file mode 100644 index ccb7767a6b693..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-18-18-04-53.bpo-27752.NEByNk.rst +++ /dev/null @@ -1 +0,0 @@ -Documentation of csv.Dialect is more descriptive. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-21-15-46-32.bpo-13814.LDcslu.rst b/Misc/NEWS.d/next/Documentation/2021-06-21-15-46-32.bpo-13814.LDcslu.rst deleted file mode 100644 index db0c6d6524bee..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-21-15-46-32.bpo-13814.LDcslu.rst +++ /dev/null @@ -1 +0,0 @@ -In the Design FAQ, answer "Why don't generators support the with statement?" diff --git a/Misc/NEWS.d/next/Documentation/2021-06-23-15-21-36.bpo-39452.o_I-6d.rst b/Misc/NEWS.d/next/Documentation/2021-06-23-15-21-36.bpo-39452.o_I-6d.rst deleted file mode 100644 index 5c8cbd8e65223..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-23-15-21-36.bpo-39452.o_I-6d.rst +++ /dev/null @@ -1,4 +0,0 @@ -Rewrote ``Doc/library/__main__.rst``. Broadened scope of the document to -explicitly discuss and differentiate between ``__main__.py`` in packages -versus the ``__name__ == '__main__'`` expression (and the idioms that -surround it). diff --git a/Misc/NEWS.d/next/Documentation/2021-06-24-14-37-16.bpo-43066.Ti7ahX.rst b/Misc/NEWS.d/next/Documentation/2021-06-24-14-37-16.bpo-43066.Ti7ahX.rst deleted file mode 100644 index 3e38522839e8b..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-24-14-37-16.bpo-43066.Ti7ahX.rst +++ /dev/null @@ -1,2 +0,0 @@ -Added a warning to :mod:`zipfile` docs: filename arg with a leading slash may cause archive to -be un-openable on Windows systems. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-26-17-41-06.bpo-40620.PAYDrB.rst b/Misc/NEWS.d/next/Documentation/2021-06-26-17-41-06.bpo-40620.PAYDrB.rst deleted file mode 100644 index 52b451b492640..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-26-17-41-06.bpo-40620.PAYDrB.rst +++ /dev/null @@ -1,2 +0,0 @@ -Convert examples in tutorial controlflow.rst section 4.3 to be interpreter-demo -style. diff --git a/Misc/NEWS.d/next/Documentation/2021-06-28-12-13-48.bpo-38062.9Ehp9O.rst b/Misc/NEWS.d/next/Documentation/2021-06-28-12-13-48.bpo-38062.9Ehp9O.rst deleted file mode 100644 index 1d90096e20bfa..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-06-28-12-13-48.bpo-38062.9Ehp9O.rst +++ /dev/null @@ -1 +0,0 @@ -Clarify that atexit uses equality comparisons internally. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-02-14-02-29.bpo-44544._5_aCz.rst b/Misc/NEWS.d/next/Documentation/2021-07-02-14-02-29.bpo-44544._5_aCz.rst deleted file mode 100644 index 4bb69977e0c1a..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-02-14-02-29.bpo-44544._5_aCz.rst +++ /dev/null @@ -1,4 +0,0 @@ -List all kwargs for :func:`textwrap.wrap`, :func:`textwrap.fill`, and -:func:`textwrap.shorten`. Now, there are nav links to attributes of -:class:`TextWrap`, which makes navigation much easier while minimizing -duplication in the documentation. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-03-18-25-17.bpo-44558.0pTknl.rst b/Misc/NEWS.d/next/Documentation/2021-07-03-18-25-17.bpo-44558.0pTknl.rst deleted file mode 100644 index a12a49ccd80cc..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-03-18-25-17.bpo-44558.0pTknl.rst +++ /dev/null @@ -1,2 +0,0 @@ -Match the docstring and python implementation of :func:`~operator.countOf` to the behavior -of its c implementation. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-12-11-39-20.bpo-44613.DIXNzc.rst b/Misc/NEWS.d/next/Documentation/2021-07-12-11-39-20.bpo-44613.DIXNzc.rst deleted file mode 100644 index baf591073620c..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-12-11-39-20.bpo-44613.DIXNzc.rst +++ /dev/null @@ -1 +0,0 @@ -importlib.metadata is no longer provisional. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-13-22-25-13.bpo-44631.qkGwe4.rst b/Misc/NEWS.d/next/Documentation/2021-07-13-22-25-13.bpo-44631.qkGwe4.rst deleted file mode 100644 index b0898fe1ad999..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-13-22-25-13.bpo-44631.qkGwe4.rst +++ /dev/null @@ -1 +0,0 @@ -Refactored the ``repr()`` code of the ``_Environ`` (os module). \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2021-07-15-11-19-03.bpo-42958.gC5IHM.rst b/Misc/NEWS.d/next/Documentation/2021-07-15-11-19-03.bpo-42958.gC5IHM.rst deleted file mode 100644 index c93b84d095526..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-15-11-19-03.bpo-42958.gC5IHM.rst +++ /dev/null @@ -1,2 +0,0 @@ -Updated the docstring and docs of :func:`filecmp.cmp` to be more accurate -and less confusing especially in respect to *shallow* arg. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-18-22-26-02.bpo-44651.SjT9iY.rst b/Misc/NEWS.d/next/Documentation/2021-07-18-22-26-02.bpo-44651.SjT9iY.rst deleted file mode 100644 index 20796e2a9bb69..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-18-22-26-02.bpo-44651.SjT9iY.rst +++ /dev/null @@ -1 +0,0 @@ -Delete entry "coercion" in Doc/glossary.rst for its outdated definition. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-18-22-43-14.bpo-44561.T7HpWm.rst b/Misc/NEWS.d/next/Documentation/2021-07-18-22-43-14.bpo-44561.T7HpWm.rst deleted file mode 100644 index 53238533edabb..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-18-22-43-14.bpo-44561.T7HpWm.rst +++ /dev/null @@ -1,3 +0,0 @@ -Update of three expired hyperlinks in Doc/distributing/index.rst: -"Project structure", "Building and packaging the project", and "Uploading the -project to the Python Packaging Index". diff --git a/Misc/NEWS.d/next/Documentation/2021-07-20-21-03-18.bpo-30511.eMFkRi.rst b/Misc/NEWS.d/next/Documentation/2021-07-20-21-03-18.bpo-30511.eMFkRi.rst deleted file mode 100644 index a358fb9cc2860..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-20-21-03-18.bpo-30511.eMFkRi.rst +++ /dev/null @@ -1,2 +0,0 @@ -Clarify that :func:`shutil.make_archive` is not thread-safe due to -reliance on changing the current working directory. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-22-08-28-03.bpo-35183.p9BWTB.rst b/Misc/NEWS.d/next/Documentation/2021-07-22-08-28-03.bpo-35183.p9BWTB.rst deleted file mode 100644 index 02c5fe82611fb..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-22-08-28-03.bpo-35183.p9BWTB.rst +++ /dev/null @@ -1 +0,0 @@ -Add typical examples to os.path.splitext docs \ No newline at end of file diff --git a/Misc/NEWS.d/next/Documentation/2021-07-25-23-04-15.bpo-44693.JuCbNq.rst b/Misc/NEWS.d/next/Documentation/2021-07-25-23-04-15.bpo-44693.JuCbNq.rst deleted file mode 100644 index 614abb412df8e..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-25-23-04-15.bpo-44693.JuCbNq.rst +++ /dev/null @@ -1,2 +0,0 @@ -Update the definition of __future__ in the glossary by replacing the confusing -word "pseudo-module" with a more accurate description. diff --git a/Misc/NEWS.d/next/Documentation/2021-07-26-23-48-31.bpo-44740.zMFGMV.rst b/Misc/NEWS.d/next/Documentation/2021-07-26-23-48-31.bpo-44740.zMFGMV.rst deleted file mode 100644 index c01273f5ddc26..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-07-26-23-48-31.bpo-44740.zMFGMV.rst +++ /dev/null @@ -1,2 +0,0 @@ -Replaced occurences of uppercase "Web" and "Internet" with lowercase -versions per the 2016 revised Associated Press Style Book. diff --git a/Misc/NEWS.d/next/Documentation/2021-08-09-19-58-45.bpo-36700.WPNW5f.rst b/Misc/NEWS.d/next/Documentation/2021-08-09-19-58-45.bpo-36700.WPNW5f.rst deleted file mode 100644 index 5bc1e23b28597..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-08-09-19-58-45.bpo-36700.WPNW5f.rst +++ /dev/null @@ -1,3 +0,0 @@ -:mod:`base64` RFC references were updated to point to :rfc:`4648`; a section -was added to point users to the new "security considerations" section of the -RFC. diff --git a/Misc/NEWS.d/next/Documentation/2021-08-11-18-02-06.bpo-33479.rCe4c5.rst b/Misc/NEWS.d/next/Documentation/2021-08-11-18-02-06.bpo-33479.rCe4c5.rst deleted file mode 100644 index c4a8a981939de..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-08-11-18-02-06.bpo-33479.rCe4c5.rst +++ /dev/null @@ -1,2 +0,0 @@ -Tkinter documentation has been greatly expanded with new "Architecture" and -"Threading model" sections. diff --git a/Misc/NEWS.d/next/Documentation/2021-08-13-19-08-03.bpo-44903.aJuvQF.rst b/Misc/NEWS.d/next/Documentation/2021-08-13-19-08-03.bpo-44903.aJuvQF.rst deleted file mode 100644 index e357405085ca0..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-08-13-19-08-03.bpo-44903.aJuvQF.rst +++ /dev/null @@ -1,3 +0,0 @@ -Removed the othergui.rst file, any references to it, and the list of GUI -frameworks in the FAQ. In their place I've added links to the Python Wiki -`page on GUI frameworks `. diff --git a/Misc/NEWS.d/next/Documentation/2021-08-13-20-17-59.bpo-16580.MZ_iK9.rst b/Misc/NEWS.d/next/Documentation/2021-08-13-20-17-59.bpo-16580.MZ_iK9.rst deleted file mode 100644 index edeca6f66e539..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-08-13-20-17-59.bpo-16580.MZ_iK9.rst +++ /dev/null @@ -1,2 +0,0 @@ -Added code equivalents for the :meth:`int.to_bytes` and :meth:`int.from_bytes` -methods, as well as tests ensuring that these code equivalents are valid. diff --git a/Misc/NEWS.d/next/Documentation/2021-08-19-15-53-08.bpo-44957.imqrh3.rst b/Misc/NEWS.d/next/Documentation/2021-08-19-15-53-08.bpo-44957.imqrh3.rst deleted file mode 100644 index 20a2aecc94ee1..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-08-19-15-53-08.bpo-44957.imqrh3.rst +++ /dev/null @@ -1,3 +0,0 @@ -Promote PEP 604 union syntax by using it where possible. Also, mention ``X | -Y`` more prominently in section about ``Union`` and mention ``X | None`` at -all in section about ``Optional``. diff --git a/Misc/NEWS.d/next/Documentation/2021-09-08-17-20-19.bpo-45024.dkNPNi.rst b/Misc/NEWS.d/next/Documentation/2021-09-08-17-20-19.bpo-45024.dkNPNi.rst deleted file mode 100644 index e73d52b8cc514..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-09-08-17-20-19.bpo-45024.dkNPNi.rst +++ /dev/null @@ -1,4 +0,0 @@ -:mod:`collections.abc` documentation has been expanded to explicitly cover -how instance and subclass checks work, with additional doctest examples and -an exhaustive list of ABCs which test membership purely by presence of the -right :term:`special method`\s. Patch by Raymond Hettinger. diff --git a/Misc/NEWS.d/next/Documentation/2021-09-18-13-45-19.bpo-45216.o56nyt.rst b/Misc/NEWS.d/next/Documentation/2021-09-18-13-45-19.bpo-45216.o56nyt.rst deleted file mode 100644 index d10b18ecdb8fd..0000000000000 --- a/Misc/NEWS.d/next/Documentation/2021-09-18-13-45-19.bpo-45216.o56nyt.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove extra documentation listing methods in ``difflib``. It was rendering -twice in pydoc and was outdated in some places. diff --git a/Misc/NEWS.d/next/IDLE/2021-05-05-09-45-24.bpo-44026.m2Z0zR.rst b/Misc/NEWS.d/next/IDLE/2021-05-05-09-45-24.bpo-44026.m2Z0zR.rst deleted file mode 100644 index bc4b680983a59..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-05-05-09-45-24.bpo-44026.m2Z0zR.rst +++ /dev/null @@ -1,2 +0,0 @@ -Include interpreter's typo fix suggestions in message line for -NameErrors and AttributeErrors. Patch by E. Paine. diff --git a/Misc/NEWS.d/next/IDLE/2021-05-09-09-02-09.bpo-44010.TaLe9x.rst b/Misc/NEWS.d/next/IDLE/2021-05-09-09-02-09.bpo-44010.TaLe9x.rst deleted file mode 100644 index becd331f6d789..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-05-09-09-02-09.bpo-44010.TaLe9x.rst +++ /dev/null @@ -1,5 +0,0 @@ -Highlight the new :ref:`match ` statement's -:ref:`soft keywords `: :keyword:`match`, -:keyword:`case `, and :keyword:`_ `. -However, this highlighting is not perfect and will be incorrect in some -rare cases, including some ``_``-s in ``case`` patterns. diff --git a/Misc/NEWS.d/next/IDLE/2021-05-27-13-39-43.bpo-41611.liNQqj.rst b/Misc/NEWS.d/next/IDLE/2021-05-27-13-39-43.bpo-41611.liNQqj.rst deleted file mode 100644 index 27d778bbe4100..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-05-27-13-39-43.bpo-41611.liNQqj.rst +++ /dev/null @@ -1 +0,0 @@ -Fix IDLE sometimes freezing upon tab-completion on macOS. diff --git a/Misc/NEWS.d/next/IDLE/2021-05-27-18-22-46.bpo-41611.jOKpfc.rst b/Misc/NEWS.d/next/IDLE/2021-05-27-18-22-46.bpo-41611.jOKpfc.rst deleted file mode 100644 index a80c9f7c5a73b..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-05-27-18-22-46.bpo-41611.jOKpfc.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid uncaught exceptions in ``AutoCompleteWindow.winconfig_event()``. diff --git a/Misc/NEWS.d/next/IDLE/2021-06-08-03-04-51.bpo-40468.tUCGUb.rst b/Misc/NEWS.d/next/IDLE/2021-06-08-03-04-51.bpo-40468.tUCGUb.rst deleted file mode 100644 index 526036ccf841e..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-06-08-03-04-51.bpo-40468.tUCGUb.rst +++ /dev/null @@ -1,4 +0,0 @@ -Split the settings dialog General tab into Windows and Shell/ED tabs. -Move help sources, which extend the Help menu, to the Extensions tab. -Make space for new options and shorten the dialog. -The latter makes the dialog better fit small screens. diff --git a/Misc/NEWS.d/next/IDLE/2021-06-10-00-50-02.bpo-33962.ikAUNg.rst b/Misc/NEWS.d/next/IDLE/2021-06-10-00-50-02.bpo-33962.ikAUNg.rst deleted file mode 100644 index b15fa8f184792..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-06-10-00-50-02.bpo-33962.ikAUNg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Move the indent space setting from the Font tab to the new Windows tab. -Patch by Mark Roseman and Terry Jan Reedy. diff --git a/Misc/NEWS.d/next/IDLE/2021-06-11-17-43-39.bpo-40128.7vDN3U.rst b/Misc/NEWS.d/next/IDLE/2021-06-11-17-43-39.bpo-40128.7vDN3U.rst deleted file mode 100644 index dafbe2cd5c3a8..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-06-11-17-43-39.bpo-40128.7vDN3U.rst +++ /dev/null @@ -1,3 +0,0 @@ -Mostly fix completions on macOS when not using tcl/tk 8.6.11 (as with 3.9). -The added update_idletask call should be harmless and possibly helpful -otherwise. diff --git a/Misc/NEWS.d/next/IDLE/2021-09-15-03-20-06.bpo-45193.G61_GV.rst b/Misc/NEWS.d/next/IDLE/2021-09-15-03-20-06.bpo-45193.G61_GV.rst deleted file mode 100644 index 94729640c71eb..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-09-15-03-20-06.bpo-45193.G61_GV.rst +++ /dev/null @@ -1 +0,0 @@ -Make completion boxes appear on Ubuntu again. diff --git a/Misc/NEWS.d/next/IDLE/2021-09-27-01-21-59.bpo-45296.9H8rdY.rst b/Misc/NEWS.d/next/IDLE/2021-09-27-01-21-59.bpo-45296.9H8rdY.rst deleted file mode 100644 index 52bade1e5327b..0000000000000 --- a/Misc/NEWS.d/next/IDLE/2021-09-27-01-21-59.bpo-45296.9H8rdY.rst +++ /dev/null @@ -1,2 +0,0 @@ -On Windows, change exit/quit message to suggest Ctrl-D, which works, instead -of , which does not work in IDLE. diff --git a/Misc/NEWS.d/next/Library/2017-09-20-14-43-03.bpo-29298._78CSN.rst b/Misc/NEWS.d/next/Library/2017-09-20-14-43-03.bpo-29298._78CSN.rst deleted file mode 100644 index e84c6de02cd27..0000000000000 --- a/Misc/NEWS.d/next/Library/2017-09-20-14-43-03.bpo-29298._78CSN.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``TypeError`` when required subparsers without ``dest`` do not receive -arguments. Patch by Anthony Sottile. diff --git a/Misc/NEWS.d/next/Library/2018-04-24-14-25-07.bpo-33349.Y_0LIr.rst b/Misc/NEWS.d/next/Library/2018-04-24-14-25-07.bpo-33349.Y_0LIr.rst deleted file mode 100644 index be68b3ea7c4a2..0000000000000 --- a/Misc/NEWS.d/next/Library/2018-04-24-14-25-07.bpo-33349.Y_0LIr.rst +++ /dev/null @@ -1 +0,0 @@ -lib2to3 now recognizes async generators everywhere. diff --git a/Misc/NEWS.d/next/Library/2019-02-26-09-31-59.bpo-26228.wyrHKc.rst b/Misc/NEWS.d/next/Library/2019-02-26-09-31-59.bpo-26228.wyrHKc.rst deleted file mode 100644 index c6ca84ae3b639..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-02-26-09-31-59.bpo-26228.wyrHKc.rst +++ /dev/null @@ -1 +0,0 @@ -pty.spawn no longer hangs on FreeBSD, macOS, and Solaris. diff --git a/Misc/NEWS.d/next/Library/2019-05-08-15-14-32.bpo-16379.rN5JVe.rst b/Misc/NEWS.d/next/Library/2019-05-08-15-14-32.bpo-16379.rN5JVe.rst deleted file mode 100644 index 874a9cf77d8c0..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-05-08-15-14-32.bpo-16379.rN5JVe.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add SQLite error code and name to :mod:`sqlite3` exceptions. -Patch by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2019-06-03-23-53-25.bpo-27513.qITN7d.rst b/Misc/NEWS.d/next/Library/2019-06-03-23-53-25.bpo-27513.qITN7d.rst deleted file mode 100644 index 90d49bb2a993f..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-06-03-23-53-25.bpo-27513.qITN7d.rst +++ /dev/null @@ -1,3 +0,0 @@ -:func:`email.utils.getaddresses` now accepts -:class:`email.header.Header` objects along with string values. -Patch by Zackery Spytz. diff --git a/Misc/NEWS.d/next/Library/2019-09-25-13-54-41.bpo-30256.wBkzox.rst b/Misc/NEWS.d/next/Library/2019-09-25-13-54-41.bpo-30256.wBkzox.rst deleted file mode 100644 index 698b0e8a61c0d..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-09-25-13-54-41.bpo-30256.wBkzox.rst +++ /dev/null @@ -1 +0,0 @@ -Pass multiprocessing BaseProxy argument ``manager_owned`` through AutoProxy. diff --git a/Misc/NEWS.d/next/Library/2019-10-08-14-08-59.bpo-38415.N1bUw6.rst b/Misc/NEWS.d/next/Library/2019-10-08-14-08-59.bpo-38415.N1bUw6.rst deleted file mode 100644 index f99bf0d19b1f8..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-10-08-14-08-59.bpo-38415.N1bUw6.rst +++ /dev/null @@ -1,3 +0,0 @@ -Added missing behavior to :func:`contextlib.asynccontextmanager` to match -:func:`contextlib.contextmanager` so decorated functions can themselves be -decorators. diff --git a/Misc/NEWS.d/next/Library/2019-11-12-18-59-33.bpo-38741.W7IYkq.rst b/Misc/NEWS.d/next/Library/2019-11-12-18-59-33.bpo-38741.W7IYkq.rst deleted file mode 100644 index 39d84ccea55b8..0000000000000 --- a/Misc/NEWS.d/next/Library/2019-11-12-18-59-33.bpo-38741.W7IYkq.rst +++ /dev/null @@ -1 +0,0 @@ -:mod:`configparser`: using ']' inside a section header will no longer cut the section name short at the ']' \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-01-16-13-54-28.bpo-39359.hzTu0h.rst b/Misc/NEWS.d/next/Library/2020-01-16-13-54-28.bpo-39359.hzTu0h.rst deleted file mode 100644 index ed4eb0c873d86..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-16-13-54-28.bpo-39359.hzTu0h.rst +++ /dev/null @@ -1 +0,0 @@ -Add one missing check that the password is a bytes object for an encrypted zipfile. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-01-16-23-41-16.bpo-38840.VzzYZz.rst b/Misc/NEWS.d/next/Library/2020-01-16-23-41-16.bpo-38840.VzzYZz.rst deleted file mode 100644 index 727f62b52a710..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-16-23-41-16.bpo-38840.VzzYZz.rst +++ /dev/null @@ -1 +0,0 @@ -Fix ``test___all__`` on platforms lacking a shared memory implementation. diff --git a/Misc/NEWS.d/next/Library/2020-01-25-12-58-20.bpo-37022.FUZI25.rst b/Misc/NEWS.d/next/Library/2020-01-25-12-58-20.bpo-37022.FUZI25.rst deleted file mode 100644 index 7b923b3aa6e44..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-01-25-12-58-20.bpo-37022.FUZI25.rst +++ /dev/null @@ -1 +0,0 @@ -:mod:`pdb` now displays exceptions from ``repr()`` with its ``p`` and ``pp`` commands. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-02-03-21-18-31.bpo-39549.l4a8uH.rst b/Misc/NEWS.d/next/Library/2020-02-03-21-18-31.bpo-39549.l4a8uH.rst deleted file mode 100644 index 91d63a96763ce..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-02-03-21-18-31.bpo-39549.l4a8uH.rst +++ /dev/null @@ -1,4 +0,0 @@ -Whereas the code for reprlib.Repr had previously used a hardcoded string -value of '...', this PR updates it to use of a ?fillvalue? attribute, whose -value defaults to '...' and can be reset in either individual reprlib.Repr -instances or in subclasses thereof. diff --git a/Misc/NEWS.d/next/Library/2020-04-24-20-39-38.bpo-34990.3SmL9M.rst b/Misc/NEWS.d/next/Library/2020-04-24-20-39-38.bpo-34990.3SmL9M.rst deleted file mode 100644 index d420b5dce1e3b..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-04-24-20-39-38.bpo-34990.3SmL9M.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed a Y2k38 bug in the compileall module where it would fail to compile -files with a modification time after the year 2038. diff --git a/Misc/NEWS.d/next/Library/2020-05-21-01-42-32.bpo-40563.fDn5bP.rst b/Misc/NEWS.d/next/Library/2020-05-21-01-42-32.bpo-40563.fDn5bP.rst deleted file mode 100644 index f20664637669a..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-05-21-01-42-32.bpo-40563.fDn5bP.rst +++ /dev/null @@ -1 +0,0 @@ -Support pathlike objects on dbm/shelve. Patch by Hakan ?elik and Henry-Joseph Aud?oud. diff --git a/Misc/NEWS.d/next/Library/2020-05-25-23-58-29.bpo-5846.O9BIfm.rst b/Misc/NEWS.d/next/Library/2020-05-25-23-58-29.bpo-5846.O9BIfm.rst deleted file mode 100644 index 556c54d0d1718..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-05-25-23-58-29.bpo-5846.O9BIfm.rst +++ /dev/null @@ -1,14 +0,0 @@ -Deprecated the following :mod:`unittest` functions, scheduled for removal in -Python 3.13: - -* :func:`~unittest.findTestCases` -* :func:`~unittest.makeSuite` -* :func:`~unittest.getTestCaseNames` - -Use :class:`~unittest.TestLoader` methods instead: - -* :meth:`unittest.TestLoader.loadTestsFromModule` -* :meth:`unittest.TestLoader.loadTestsFromTestCase` -* :meth:`unittest.TestLoader.getTestCaseNames` - -Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2020-05-30-10-48-04.bpo-24391.ZCTnhX.rst b/Misc/NEWS.d/next/Library/2020-05-30-10-48-04.bpo-24391.ZCTnhX.rst deleted file mode 100644 index 15add1531501a..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-05-30-10-48-04.bpo-24391.ZCTnhX.rst +++ /dev/null @@ -1,3 +0,0 @@ -Improved reprs of :mod:`threading` synchronization objects: -:class:`~threading.Semaphore`, :class:`~threading.BoundedSemaphore`, -:class:`~threading.Event` and :class:`~threading.Barrier`. diff --git a/Misc/NEWS.d/next/Library/2020-07-01-17-42-41.bpo-41137.AnqbP-.rst b/Misc/NEWS.d/next/Library/2020-07-01-17-42-41.bpo-41137.AnqbP-.rst deleted file mode 100644 index f91b47dd72461..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-01-17-42-41.bpo-41137.AnqbP-.rst +++ /dev/null @@ -1 +0,0 @@ -Use utf-8 encoding while reading .pdbrc files. Patch by Srinivas Reddy Thatiparthy \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2020-07-13-23-46-59.bpo-32695.tTqqXe.rst b/Misc/NEWS.d/next/Library/2020-07-13-23-46-59.bpo-32695.tTqqXe.rst deleted file mode 100644 index c71316ed656a2..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-13-23-46-59.bpo-32695.tTqqXe.rst +++ /dev/null @@ -1,2 +0,0 @@ -The *compresslevel* and *preset* keyword arguments of :func:`tarfile.open` -are now both documented and tested. diff --git a/Misc/NEWS.d/next/Library/2020-07-26-18-17-30.bpo-41402.YRkVkp.rst b/Misc/NEWS.d/next/Library/2020-07-26-18-17-30.bpo-41402.YRkVkp.rst deleted file mode 100644 index 45585a469e724..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-26-18-17-30.bpo-41402.YRkVkp.rst +++ /dev/null @@ -1 +0,0 @@ -Fix :meth:`email.message.EmailMessage.set_content` when called with binary data and ``7bit`` content transfer encoding. diff --git a/Misc/NEWS.d/next/Library/2020-07-30-14-37-15.bpo-20684.qV35GU.rst b/Misc/NEWS.d/next/Library/2020-07-30-14-37-15.bpo-20684.qV35GU.rst deleted file mode 100644 index 56bc1e4cb4ef0..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-07-30-14-37-15.bpo-20684.qV35GU.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove unused ``_signature_get_bound_param`` function from :mod:`inspect` - -by Anthony Sottile. diff --git a/Misc/NEWS.d/next/Library/2020-09-10-07-23-24.bpo-41730.DyKFi9.rst b/Misc/NEWS.d/next/Library/2020-09-10-07-23-24.bpo-41730.DyKFi9.rst deleted file mode 100644 index 63d8353a7aab2..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-09-10-07-23-24.bpo-41730.DyKFi9.rst +++ /dev/null @@ -1 +0,0 @@ -``DeprecationWarning`` is now raised when importing :mod:`tkinter.tix`, which has been deprecated in documentation since Python 3.6. diff --git a/Misc/NEWS.d/next/Library/2020-10-01-21-46-34.bpo-40956._tvsZ7.rst b/Misc/NEWS.d/next/Library/2020-10-01-21-46-34.bpo-40956._tvsZ7.rst deleted file mode 100644 index adec299d2316a..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-01-21-46-34.bpo-40956._tvsZ7.rst +++ /dev/null @@ -1 +0,0 @@ -Use Argument Clinic in :mod:`sqlite3`. Patches by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2020-10-11-20-23-48.bpo-37449.f-t3V6.rst b/Misc/NEWS.d/next/Library/2020-10-11-20-23-48.bpo-37449.f-t3V6.rst deleted file mode 100644 index 2202ae0a9ac96..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-11-20-23-48.bpo-37449.f-t3V6.rst +++ /dev/null @@ -1 +0,0 @@ -``ensurepip`` now uses ``importlib.resources.files()`` traversable APIs diff --git a/Misc/NEWS.d/next/Library/2020-10-18-09-42-53.bpo-40497.CRz2sG.rst b/Misc/NEWS.d/next/Library/2020-10-18-09-42-53.bpo-40497.CRz2sG.rst deleted file mode 100644 index 067c48626c308..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-10-18-09-42-53.bpo-40497.CRz2sG.rst +++ /dev/null @@ -1,4 +0,0 @@ -:meth:`subprocess.check_output` now raises :exc:`ValueError` when the -invalid keyword argument *check* is passed by user code. Previously -such use would fail later with a :exc:`TypeError`. -Patch by R?mi Lapeyre. diff --git a/Misc/NEWS.d/next/Library/2020-12-08-01-08-58.bpo-41818.zO8vV7.rst b/Misc/NEWS.d/next/Library/2020-12-08-01-08-58.bpo-41818.zO8vV7.rst deleted file mode 100644 index e8d6063d4b5e7..0000000000000 --- a/Misc/NEWS.d/next/Library/2020-12-08-01-08-58.bpo-41818.zO8vV7.rst +++ /dev/null @@ -1 +0,0 @@ -Soumendra Ganguly: add termios.tcgetwinsize(), termios.tcsetwinsize(). \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-01-13-00-02-44.bpo-42862.Z6ACLN.rst b/Misc/NEWS.d/next/Library/2021-01-13-00-02-44.bpo-42862.Z6ACLN.rst deleted file mode 100644 index beda25fd335b0..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-13-00-02-44.bpo-42862.Z6ACLN.rst +++ /dev/null @@ -1,4 +0,0 @@ -:mod:`sqlite3` now utilizes :meth:`functools.lru_cache` to implement the -connection statement cache. As a small optimisation, the default -statement cache size has been increased from 100 to 128. -Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-01-16-18-36-00.bpo-33809.BiMK6V.rst b/Misc/NEWS.d/next/Library/2021-01-16-18-36-00.bpo-33809.BiMK6V.rst deleted file mode 100644 index a8a550dc0d418..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-16-18-36-00.bpo-33809.BiMK6V.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add the :meth:`traceback.TracebackException.print` method which prints -the formatted exception information. diff --git a/Misc/NEWS.d/next/Library/2021-01-25-21-24-55.bpo-43024.vAUrIi.rst b/Misc/NEWS.d/next/Library/2021-01-25-21-24-55.bpo-43024.vAUrIi.rst deleted file mode 100644 index 56596ce3c2a97..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-25-21-24-55.bpo-43024.vAUrIi.rst +++ /dev/null @@ -1 +0,0 @@ -Improve the help signature of :func:`traceback.print_exception`, :func:`traceback.format_exception` and :func:`traceback.format_exception_only`. diff --git a/Misc/NEWS.d/next/Library/2021-01-31-18-24-54.bpo-43086.2_P-SH.rst b/Misc/NEWS.d/next/Library/2021-01-31-18-24-54.bpo-43086.2_P-SH.rst deleted file mode 100644 index f49e7a84cc537..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-01-31-18-24-54.bpo-43086.2_P-SH.rst +++ /dev/null @@ -1,3 +0,0 @@ -Added a new optional :code:`strict_mode` parameter to *binascii.a2b_base64*. -When :code:`scrict_mode` is set to :code:`True`, the *a2b_base64* function will accept only valid base64 content. -More details about what "valid base64 content" is, can be found in the function's documentation. diff --git a/Misc/NEWS.d/next/Library/2021-02-02-20-11-14.bpo-42971.OpVoFu.rst b/Misc/NEWS.d/next/Library/2021-02-02-20-11-14.bpo-42971.OpVoFu.rst deleted file mode 100644 index 97c8d2d79aa40..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-02-20-11-14.bpo-42971.OpVoFu.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add definition of ``errno.EQFULL`` for platforms that define this constant -(such as macOS). diff --git a/Misc/NEWS.d/next/Library/2021-02-04-23-16-03.bpo-30077.v6TqAi.rst b/Misc/NEWS.d/next/Library/2021-02-04-23-16-03.bpo-30077.v6TqAi.rst deleted file mode 100644 index 4af17eed8f733..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-04-23-16-03.bpo-30077.v6TqAi.rst +++ /dev/null @@ -1 +0,0 @@ -Added support for Apple's aifc/sowt pseudo-compression \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-02-15-21-17-46.bpo-43232.awc4yZ.rst b/Misc/NEWS.d/next/Library/2021-02-15-21-17-46.bpo-43232.awc4yZ.rst deleted file mode 100644 index a527a7ba95657..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-15-21-17-46.bpo-43232.awc4yZ.rst +++ /dev/null @@ -1,2 +0,0 @@ -Prohibit previously deprecated potentially disruptive operations on -:class:`asyncio.trsock.TransportSocket`. Patch by Illia Volochii. diff --git a/Misc/NEWS.d/next/Library/2021-02-15-22-14-31.bpo-43234.F-vKAT.rst b/Misc/NEWS.d/next/Library/2021-02-15-22-14-31.bpo-43234.F-vKAT.rst deleted file mode 100644 index 7f195cc752fc1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-15-22-14-31.bpo-43234.F-vKAT.rst +++ /dev/null @@ -1,3 +0,0 @@ -Prohibit passing non-:class:`concurrent.futures.ThreadPoolExecutor` -executors to :meth:`loop.set_default_executor` following a deprecation in -Python 3.8. Patch by Illia Volochii. diff --git a/Misc/NEWS.d/next/Library/2021-02-25-08-32-06.bpo-43318.bZJw6V.rst b/Misc/NEWS.d/next/Library/2021-02-25-08-32-06.bpo-43318.bZJw6V.rst deleted file mode 100644 index c2c9c8776fd86..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-02-25-08-32-06.bpo-43318.bZJw6V.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a bug where :mod:`pdb` does not always echo cleared breakpoints. diff --git a/Misc/NEWS.d/next/Library/2021-03-03-13-32-37.bpo-43392.QQumou.rst b/Misc/NEWS.d/next/Library/2021-03-03-13-32-37.bpo-43392.QQumou.rst deleted file mode 100644 index 175836b89170a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-03-03-13-32-37.bpo-43392.QQumou.rst +++ /dev/null @@ -1,4 +0,0 @@ -:func:`importlib._bootstrap._find_and_load` now implements a two-step -check to avoid locking when modules have been already imported and are -ready. This improves performance of repeated calls to -:func:`importlib.import_module` and :func:`importlib.__import__`. diff --git a/Misc/NEWS.d/next/Library/2021-03-24-09-40-02.bpo-43612.vMGZ4y.rst b/Misc/NEWS.d/next/Library/2021-03-24-09-40-02.bpo-43612.vMGZ4y.rst deleted file mode 100644 index e6fc88f45eea5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-03-24-09-40-02.bpo-43612.vMGZ4y.rst +++ /dev/null @@ -1,5 +0,0 @@ -:func:`zlib.compress` now accepts a wbits parameter which allows users to -compress data as a raw deflate block without zlib headers and trailers in -one go. Previously this required instantiating a ``zlib.compressobj``. It -also provides a faster alternative to ``gzip.compress`` when wbits=31 is -used. diff --git a/Misc/NEWS.d/next/Library/2021-03-29-00-23-30.bpo-43650.v01tic.rst b/Misc/NEWS.d/next/Library/2021-03-29-00-23-30.bpo-43650.v01tic.rst deleted file mode 100644 index a2ea4a4800a73..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-03-29-00-23-30.bpo-43650.v01tic.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix :exc:`MemoryError` in :func:`shutil.unpack_archive` which fails inside -:func:`shutil._unpack_zipfile` on large files. Patch by Igor Bolshakov. diff --git a/Misc/NEWS.d/next/Library/2021-03-30-08-39-08.bpo-43666.m72tlH.rst b/Misc/NEWS.d/next/Library/2021-03-30-08-39-08.bpo-43666.m72tlH.rst deleted file mode 100644 index 6a3432191d61b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-03-30-08-39-08.bpo-43666.m72tlH.rst +++ /dev/null @@ -1,6 +0,0 @@ -AIX: `Lib/_aix_support.get_platform()` may fail in an AIX WPAR. -The fileset bos.rte appears to have a builddate in both LPAR and WPAR -so this fileset is queried rather than bos.mp64. -To prevent a similiar situation (no builddate in ODM) a value (9988) -sufficient for completing a build is provided. -Patch by M Felt. diff --git a/Misc/NEWS.d/next/Library/2021-04-15-12-02-17.bpo-43853.XXCVAp.rst b/Misc/NEWS.d/next/Library/2021-04-15-12-02-17.bpo-43853.XXCVAp.rst deleted file mode 100644 index a0164c4665a05..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-04-15-12-02-17.bpo-43853.XXCVAp.rst +++ /dev/null @@ -1,7 +0,0 @@ -Improved string handling for :mod:`sqlite3` user-defined functions and -aggregates: - -* It is now possible to pass strings with embedded null characters to UDFs -* Conversion failures now correctly raise :exc:`MemoryError` - -Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-04-29-00-48-00.bpo-28528.JLAVWj.rst b/Misc/NEWS.d/next/Library/2021-04-29-00-48-00.bpo-28528.JLAVWj.rst deleted file mode 100644 index 97731ad882e03..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-04-29-00-48-00.bpo-28528.JLAVWj.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a bug in :mod:`pdb` where :meth:`~pdb.Pdb.checkline` raises -:exc:`AttributeError` if it is called after :meth:`~pdb.Pdb.reset`. diff --git a/Misc/NEWS.d/next/Library/2021-04-30-16-58-24.bpo-43972.Y2r9lg.rst b/Misc/NEWS.d/next/Library/2021-04-30-16-58-24.bpo-43972.Y2r9lg.rst deleted file mode 100644 index 3d67b885bab10..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-04-30-16-58-24.bpo-43972.Y2r9lg.rst +++ /dev/null @@ -1,3 +0,0 @@ -When :class:`http.server.SimpleHTTPRequestHandler` sends a -``301 (Moved Permanently)`` for a directory path not ending with `/`, add a -``Content-Length: 0`` header. This improves the behavior for certain clients. diff --git a/Misc/NEWS.d/next/Library/2021-05-01-15-43-37.bpo-44002.KLT_wd.rst b/Misc/NEWS.d/next/Library/2021-05-01-15-43-37.bpo-44002.KLT_wd.rst deleted file mode 100644 index 9d662d9827a91..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-01-15-43-37.bpo-44002.KLT_wd.rst +++ /dev/null @@ -1,5 +0,0 @@ -:mod:`urllib.parse` now uses :func:`functool.lru_cache` for its internal URL -splitting and quoting caches instead of rolling its own like its the '90s. - -The undocumented internal :mod:`urllib.parse` ``Quoted`` class API is now -deprecated, for removal in 3.14. diff --git a/Misc/NEWS.d/next/Library/2021-05-02-13-54-25.bpo-38352.N9MlhV.rst b/Misc/NEWS.d/next/Library/2021-05-02-13-54-25.bpo-38352.N9MlhV.rst deleted file mode 100644 index bf8fe758f3570..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-02-13-54-25.bpo-38352.N9MlhV.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ``IO``, ``BinaryIO``, ``TextIO``, ``Match``, and ``Pattern`` to -``typing.__all__``. Patch by Jelle Zijlstra. diff --git a/Misc/NEWS.d/next/Library/2021-05-03-10-07-43.bpo-44018.VDyW8f.rst b/Misc/NEWS.d/next/Library/2021-05-03-10-07-43.bpo-44018.VDyW8f.rst deleted file mode 100644 index 87c7d83a7f35c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-03-10-07-43.bpo-44018.VDyW8f.rst +++ /dev/null @@ -1 +0,0 @@ -random.seed() no longer mutates bytearray inputs. diff --git a/Misc/NEWS.d/next/Library/2021-05-03-19-59-14.bpo-40465.1tB4Y0.rst b/Misc/NEWS.d/next/Library/2021-05-03-19-59-14.bpo-40465.1tB4Y0.rst deleted file mode 100644 index b8b63debdbc19..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-03-19-59-14.bpo-40465.1tB4Y0.rst +++ /dev/null @@ -1 +0,0 @@ -Remove random module features deprecated in Python 3.9. diff --git a/Misc/NEWS.d/next/Library/2021-05-05-11-44-49.bpo-36515.uOSa3q.rst b/Misc/NEWS.d/next/Library/2021-05-05-11-44-49.bpo-36515.uOSa3q.rst deleted file mode 100644 index dd24474c2fde7..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-05-11-44-49.bpo-36515.uOSa3q.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :mod:`hashlib` module no longer does unaligned memory accesses when -compiled for ARM platforms. diff --git a/Misc/NEWS.d/next/Library/2021-05-06-16-01-55.bpo-44059.GF5r6O.rst b/Misc/NEWS.d/next/Library/2021-05-06-16-01-55.bpo-44059.GF5r6O.rst deleted file mode 100644 index f734bdb0bce09..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-06-16-01-55.bpo-44059.GF5r6O.rst +++ /dev/null @@ -1 +0,0 @@ -Register the SerenityOS Browser in the :mod:`webbrowser` module. diff --git a/Misc/NEWS.d/next/Library/2021-05-07-08-39-23.bpo-44061.MvElG6.rst b/Misc/NEWS.d/next/Library/2021-05-07-08-39-23.bpo-44061.MvElG6.rst deleted file mode 100644 index e41f285fae949..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-07-08-39-23.bpo-44061.MvElG6.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix regression in previous release when calling :func:`pkgutil.iter_modules` -with a list of :class:`pathlib.Path` objects diff --git a/Misc/NEWS.d/next/Library/2021-05-09-03-26-31.bpo-44081.A-Mrto.rst b/Misc/NEWS.d/next/Library/2021-05-09-03-26-31.bpo-44081.A-Mrto.rst deleted file mode 100644 index e4a09e366bd80..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-09-03-26-31.bpo-44081.A-Mrto.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`ast.unparse` now doesn't use redundant spaces to separate ``lambda`` -and the ``:`` if there are no parameters. diff --git a/Misc/NEWS.d/next/Library/2021-05-09-22-52-34.bpo-44089.IoANsN.rst b/Misc/NEWS.d/next/Library/2021-05-09-22-52-34.bpo-44089.IoANsN.rst deleted file mode 100644 index b9bd963582fdc..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-09-22-52-34.bpo-44089.IoANsN.rst +++ /dev/null @@ -1,2 +0,0 @@ -Allow subclassing ``csv.Error`` in 3.10 (it was allowed in 3.9 and earlier but -was disallowed in early versions of 3.10). diff --git a/Misc/NEWS.d/next/Library/2021-05-10-17-45-00.bpo-44098._MoxuZ.rst b/Misc/NEWS.d/next/Library/2021-05-10-17-45-00.bpo-44098._MoxuZ.rst deleted file mode 100644 index 2aaa8b695af46..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-10-17-45-00.bpo-44098._MoxuZ.rst +++ /dev/null @@ -1,5 +0,0 @@ -``typing.ParamSpec`` will no longer be found in the ``__parameters__`` of -most :mod:`typing` generics except in valid use locations specified by -:pep:`612`. This prevents incorrect usage like ``typing.List[P][int]``. This -change means incorrect usage which may have passed silently in 3.10 beta 1 -and earlier will now error. diff --git a/Misc/NEWS.d/next/Library/2021-05-12-16-43-21.bpo-38908.nM2_rO.rst b/Misc/NEWS.d/next/Library/2021-05-12-16-43-21.bpo-38908.nM2_rO.rst deleted file mode 100644 index 18e3dd4066c4a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-12-16-43-21.bpo-38908.nM2_rO.rst +++ /dev/null @@ -1,5 +0,0 @@ -Subclasses of ``typing.Protocol`` which only have data variables declared -will now raise a ``TypeError`` when checked with ``isinstance`` unless they -are decorated with :func:`runtime_checkable`. Previously, these checks -passed silently. -Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-05-13-19-07-28.bpo-37788.adeFcf.rst b/Misc/NEWS.d/next/Library/2021-05-13-19-07-28.bpo-37788.adeFcf.rst deleted file mode 100644 index 0c33923e99245..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-13-19-07-28.bpo-37788.adeFcf.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a reference leak when a Thread object is never joined. diff --git a/Misc/NEWS.d/next/Library/2021-05-13-19-44-38.bpo-44077.04b2a4.rst b/Misc/NEWS.d/next/Library/2021-05-13-19-44-38.bpo-44077.04b2a4.rst deleted file mode 100644 index 7bb4379f571b6..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-13-19-44-38.bpo-44077.04b2a4.rst +++ /dev/null @@ -1,3 +0,0 @@ -It's now possible to receive the type of service (ToS), a.k.a. differentiated -services (DS), a.k.a. differenciated services code point (DSCP) and excplicit -congestion notification (ECN) IP header fields with ``socket.IP_RECVTOS``. diff --git a/Misc/NEWS.d/next/Library/2021-05-14-16-06-02.bpo-44095.v_pLwY.rst b/Misc/NEWS.d/next/Library/2021-05-14-16-06-02.bpo-44095.v_pLwY.rst deleted file mode 100644 index ee03e933f35d6..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-14-16-06-02.bpo-44095.v_pLwY.rst +++ /dev/null @@ -1,2 +0,0 @@ -:class:`zipfile.Path` now supports :attr:`zipfile.Path.stem`, -:attr:`zipfile.Path.suffixes`, and :attr:`zipfile.Path.suffix` attributes. diff --git a/Misc/NEWS.d/next/Library/2021-05-16-00-00-38.bpo-44145.ko5SJ7.rst b/Misc/NEWS.d/next/Library/2021-05-16-00-00-38.bpo-44145.ko5SJ7.rst deleted file mode 100644 index 40222185d5067..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-16-00-00-38.bpo-44145.ko5SJ7.rst +++ /dev/null @@ -1,3 +0,0 @@ -:mod:`hmac` computations were not releasing the GIL while calling the -OpenSSL ``HMAC_Update`` C API (a new feature in 3.9). This unintentionally -prevented parallel computation as other :mod:`hashlib` algorithms support. diff --git a/Misc/NEWS.d/next/Library/2021-05-16-02-24-23.bpo-44142.t-XU8k.rst b/Misc/NEWS.d/next/Library/2021-05-16-02-24-23.bpo-44142.t-XU8k.rst deleted file mode 100644 index 96fdd7c6566b2..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-16-02-24-23.bpo-44142.t-XU8k.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`ast.unparse` will now drop the redundant parentheses when tuples used -as assignment targets (e.g in for loops). diff --git a/Misc/NEWS.d/next/Library/2021-05-16-11-57-38.bpo-44150.xAhhik.rst b/Misc/NEWS.d/next/Library/2021-05-16-11-57-38.bpo-44150.xAhhik.rst deleted file mode 100644 index f4c2786d13b05..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-16-11-57-38.bpo-44150.xAhhik.rst +++ /dev/null @@ -1 +0,0 @@ -Add optional *weights* argument to statistics.fmean(). diff --git a/Misc/NEWS.d/next/Library/2021-05-16-17-48-24.bpo-33433.MyzO71.rst b/Misc/NEWS.d/next/Library/2021-05-16-17-48-24.bpo-33433.MyzO71.rst deleted file mode 100644 index 703e038fac985..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-16-17-48-24.bpo-33433.MyzO71.rst +++ /dev/null @@ -1 +0,0 @@ -For IPv4 mapped IPv6 addresses (:rfc:`4291` Section 2.5.5.2), the :mod:`ipaddress.IPv6Address.is_private` check is deferred to the mapped IPv4 address. This solves a bug where public mapped IPv4 addresses were considered private by the IPv6 check. diff --git a/Misc/NEWS.d/next/Library/2021-05-17-07-24-24.bpo-44154.GRI5bf.rst b/Misc/NEWS.d/next/Library/2021-05-17-07-24-24.bpo-44154.GRI5bf.rst deleted file mode 100644 index 3ec326e875ecc..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-17-07-24-24.bpo-44154.GRI5bf.rst +++ /dev/null @@ -1 +0,0 @@ -Optimize :class:`fractions.Fraction` pickling for large components. diff --git a/Misc/NEWS.d/next/Library/2021-05-17-21-05-06.bpo-4928.Ot2yjO.rst b/Misc/NEWS.d/next/Library/2021-05-17-21-05-06.bpo-4928.Ot2yjO.rst deleted file mode 100644 index 359f801582154..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-17-21-05-06.bpo-4928.Ot2yjO.rst +++ /dev/null @@ -1 +0,0 @@ -Documented existing behavior on POSIX: NamedTemporaryFiles are not deleted when creating process is killed with SIGKILL \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-05-18-00-17-21.bpo-27334.32EJZi.rst b/Misc/NEWS.d/next/Library/2021-05-18-00-17-21.bpo-27334.32EJZi.rst deleted file mode 100644 index dc0cdf33ec5ac..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-18-00-17-21.bpo-27334.32EJZi.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :mod:`sqlite3` context manager now performs a rollback (thus releasing the -database lock) if commit failed. Patch by Luca Citi and Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-05-21-12-12-35.bpo-43643.GWnmcF.rst b/Misc/NEWS.d/next/Library/2021-05-21-12-12-35.bpo-43643.GWnmcF.rst deleted file mode 100644 index 57157dfe217ee..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-21-12-12-35.bpo-43643.GWnmcF.rst +++ /dev/null @@ -1 +0,0 @@ -Declare readers.MultiplexedPath.name as a property per the spec. diff --git a/Misc/NEWS.d/next/Library/2021-05-21-21-23-43.bpo-44210.5afQ3K.rst b/Misc/NEWS.d/next/Library/2021-05-21-21-23-43.bpo-44210.5afQ3K.rst deleted file mode 100644 index e5a14a1a265f1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-21-21-23-43.bpo-44210.5afQ3K.rst +++ /dev/null @@ -1 +0,0 @@ -Make importlib.metadata._meta.PackageMetadata public. diff --git a/Misc/NEWS.d/next/Library/2021-05-25-23-26-38.bpo-43216.xTUyyX.rst b/Misc/NEWS.d/next/Library/2021-05-25-23-26-38.bpo-43216.xTUyyX.rst deleted file mode 100644 index 845ef95d1ad29..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-25-23-26-38.bpo-43216.xTUyyX.rst +++ /dev/null @@ -1,6 +0,0 @@ -Remove the :func:`@asyncio.coroutine ` :term:`decorator` -enabling legacy generator-based coroutines to be compatible with async/await -code; remove :class:`asyncio.coroutines.CoroWrapper` used for wrapping -legacy coroutine objects in the debug mode. The decorator has been deprecated -since Python 3.8 and the removal was initially scheduled for Python 3.10. -Patch by Illia Volochii. diff --git a/Misc/NEWS.d/next/Library/2021-05-26-13-15-51.bpo-44241.TBqej8.rst b/Misc/NEWS.d/next/Library/2021-05-26-13-15-51.bpo-44241.TBqej8.rst deleted file mode 100644 index c160cf70abb52..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-26-13-15-51.bpo-44241.TBqej8.rst +++ /dev/null @@ -1,2 +0,0 @@ -Incorporate minor tweaks from importlib_metadata 4.1: SimplePath protocol, -support for Metadata 2.2. diff --git a/Misc/NEWS.d/next/Library/2021-05-26-13-34-37.bpo-33693.3okzdo.rst b/Misc/NEWS.d/next/Library/2021-05-26-13-34-37.bpo-33693.3okzdo.rst deleted file mode 100644 index 2a568a4f469f8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-26-13-34-37.bpo-33693.3okzdo.rst +++ /dev/null @@ -1 +0,0 @@ -Importlib.metadata now prefers f-strings to .format. diff --git a/Misc/NEWS.d/next/Library/2021-05-26-14-50-06.bpo-38693.NkMacJ.rst b/Misc/NEWS.d/next/Library/2021-05-26-14-50-06.bpo-38693.NkMacJ.rst deleted file mode 100644 index 10d014b5da562..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-26-14-50-06.bpo-38693.NkMacJ.rst +++ /dev/null @@ -1 +0,0 @@ -Prefer f-strings to ``.format`` in importlib.resources. diff --git a/Misc/NEWS.d/next/Library/2021-05-26-22-04-40.bpo-44235.qFBYpp.rst b/Misc/NEWS.d/next/Library/2021-05-26-22-04-40.bpo-44235.qFBYpp.rst deleted file mode 100644 index 41af18175d95b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-26-22-04-40.bpo-44235.qFBYpp.rst +++ /dev/null @@ -1 +0,0 @@ -Remove deprecated functions in the :mod:`gettext`. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-05-28-09-43-33.bpo-44258.nh5F7R.rst b/Misc/NEWS.d/next/Library/2021-05-28-09-43-33.bpo-44258.nh5F7R.rst deleted file mode 100644 index b9636899700f6..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-28-09-43-33.bpo-44258.nh5F7R.rst +++ /dev/null @@ -1 +0,0 @@ -Support PEP 515 for Fraction's initialization from string. diff --git a/Misc/NEWS.d/next/Library/2021-05-29-01-05-43.bpo-44254.f06xDm.rst b/Misc/NEWS.d/next/Library/2021-05-29-01-05-43.bpo-44254.f06xDm.rst deleted file mode 100644 index 7438d5ce044b8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-29-01-05-43.bpo-44254.f06xDm.rst +++ /dev/null @@ -1,2 +0,0 @@ -On Mac, give turtledemo button text a color that works on both light -or dark background. Programmers cannot control the latter. diff --git a/Misc/NEWS.d/next/Library/2021-05-30-13-32-09.bpo-44260.ROEbVd.rst b/Misc/NEWS.d/next/Library/2021-05-30-13-32-09.bpo-44260.ROEbVd.rst deleted file mode 100644 index a63af627e6fbf..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-30-13-32-09.bpo-44260.ROEbVd.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :class:`random.Random` constructor no longer reads system entropy -without need. diff --git a/Misc/NEWS.d/next/Library/2021-05-31-04-51-02.bpo-43858.r7LOu6.rst b/Misc/NEWS.d/next/Library/2021-05-31-04-51-02.bpo-43858.r7LOu6.rst deleted file mode 100644 index d864e1b4e51e3..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-31-04-51-02.bpo-43858.r7LOu6.rst +++ /dev/null @@ -1 +0,0 @@ -Added a function that returns a copy of a dict of logging levels: :func:`logging.getLevelNamesMapping` diff --git a/Misc/NEWS.d/next/Library/2021-05-31-11-28-03.bpo-44246.nhmt-v.rst b/Misc/NEWS.d/next/Library/2021-05-31-11-28-03.bpo-44246.nhmt-v.rst deleted file mode 100644 index 727d9fd0a19d8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-31-11-28-03.bpo-44246.nhmt-v.rst +++ /dev/null @@ -1,3 +0,0 @@ -In importlib.metadata.entry_points, de-duplication of distributions no -longer requires loading the full metadata for PathDistribution objects, -improving entry point loading performance by ~10x. diff --git a/Misc/NEWS.d/next/Library/2021-05-31-11-34-56.bpo-44246.yHAkF0.rst b/Misc/NEWS.d/next/Library/2021-05-31-11-34-56.bpo-44246.yHAkF0.rst deleted file mode 100644 index b93f8b02dc476..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-05-31-11-34-56.bpo-44246.yHAkF0.rst +++ /dev/null @@ -1,7 +0,0 @@ -In ``importlib.metadata``, restore compatibility in the result from -``Distribution.entry_points`` (``EntryPoints``) to honor expectations in -older implementations and issuing deprecation warnings for these cases: A. ``EntryPoints`` objects are once again mutable, allowing for ``sort()`` -and other list-based mutation operations. Avoid deprecation warnings by -casting to a mutable sequence (e.g. ``list(dist.entry_points).sort()``). B. ``EntryPoints`` results once again allow for access by index. To avoid -deprecation warnings, cast the result to a Sequence first (e.g. -``tuple(dist.entry_points)[0]``). diff --git a/Misc/NEWS.d/next/Library/2021-06-07-10-26-14.bpo-44242.MKeMCQ.rst b/Misc/NEWS.d/next/Library/2021-06-07-10-26-14.bpo-44242.MKeMCQ.rst deleted file mode 100644 index 39740b6736591..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-07-10-26-14.bpo-44242.MKeMCQ.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove missing flag check from Enum creation and move into a ``verify`` -decorator. diff --git a/Misc/NEWS.d/next/Library/2021-06-08-17-47-38.bpo-44339.9JwMSc.rst b/Misc/NEWS.d/next/Library/2021-06-08-17-47-38.bpo-44339.9JwMSc.rst deleted file mode 100644 index 10499eb02bb3c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-08-17-47-38.bpo-44339.9JwMSc.rst +++ /dev/null @@ -1,3 +0,0 @@ -Change ``math.pow(?0.0, -math.inf)`` to return ``inf`` instead of raising -``ValueError``. This brings the special-case handling of ``math.pow`` into -compliance with the IEEE 754 standard. diff --git a/Misc/NEWS.d/next/Library/2021-06-09-08-32-39.bpo-44357.70Futb.rst b/Misc/NEWS.d/next/Library/2021-06-09-08-32-39.bpo-44357.70Futb.rst deleted file mode 100644 index f169a464f9fe7..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-09-08-32-39.bpo-44357.70Futb.rst +++ /dev/null @@ -1 +0,0 @@ -Added a function that returns cube root of the given number :func:`math.cbrt` \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-06-09-10-08-32.bpo-35800.3hmkWw.rst b/Misc/NEWS.d/next/Library/2021-06-09-10-08-32.bpo-35800.3hmkWw.rst deleted file mode 100644 index d3bf596b75028..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-09-10-08-32.bpo-35800.3hmkWw.rst +++ /dev/null @@ -1,2 +0,0 @@ -:class:`smtpd.MailmanProxy` is now removed as it is unusable without an -external module, ``mailman``. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-06-10-07-26-12.bpo-44351.rvyf2v.rst b/Misc/NEWS.d/next/Library/2021-06-10-07-26-12.bpo-44351.rvyf2v.rst deleted file mode 100644 index d731a549632b5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-10-07-26-12.bpo-44351.rvyf2v.rst +++ /dev/null @@ -1,2 +0,0 @@ -Restore back :func:`parse_makefile` in :mod:`distutils.sysconfig` because it -behaves differently than the similar implementation in :mod:`sysconfig`. diff --git a/Misc/NEWS.d/next/Library/2021-06-10-08-35-38.bpo-44356.6oDFhO.rst b/Misc/NEWS.d/next/Library/2021-06-10-08-35-38.bpo-44356.6oDFhO.rst deleted file mode 100644 index 954a803fe25c1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-10-08-35-38.bpo-44356.6oDFhO.rst +++ /dev/null @@ -1 +0,0 @@ -[Enum] Allow multiple data-type mixins if they are all the same. diff --git a/Misc/NEWS.d/next/Library/2021-06-10-15-06-47.bpo-44342.qqkGlj.rst b/Misc/NEWS.d/next/Library/2021-06-10-15-06-47.bpo-44342.qqkGlj.rst deleted file mode 100644 index 6db75e3e9bcf1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-10-15-06-47.bpo-44342.qqkGlj.rst +++ /dev/null @@ -1 +0,0 @@ -[Enum] Change pickling from by-value to by-name. diff --git a/Misc/NEWS.d/next/Library/2021-06-10-20-07-32.bpo-44362.oVOMfd.rst b/Misc/NEWS.d/next/Library/2021-06-10-20-07-32.bpo-44362.oVOMfd.rst deleted file mode 100644 index 0e6aef3c90e6f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-10-20-07-32.bpo-44362.oVOMfd.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improve :mod:`ssl` module's deprecation messages, error reporting, and -documentation for deprecations. diff --git a/Misc/NEWS.d/next/Library/2021-06-10-21-53-46.bpo-34266.k3fxnm.rst b/Misc/NEWS.d/next/Library/2021-06-10-21-53-46.bpo-34266.k3fxnm.rst deleted file mode 100644 index 22ef84e9626ad..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-10-21-53-46.bpo-34266.k3fxnm.rst +++ /dev/null @@ -1 +0,0 @@ -Handle exceptions from parsing the arg of :mod:`pdb`'s run/restart command. diff --git a/Misc/NEWS.d/next/Library/2021-06-12-10-08-14.bpo-44395.PcW6Sx.rst b/Misc/NEWS.d/next/Library/2021-06-12-10-08-14.bpo-44395.PcW6Sx.rst deleted file mode 100644 index 6172eec0a9bd3..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-12-10-08-14.bpo-44395.PcW6Sx.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix :meth:`~email.message.MIMEPart.as_string` to pass unixfrom properly. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-06-12-21-25-35.bpo-27827.TMWh1i.rst b/Misc/NEWS.d/next/Library/2021-06-12-21-25-35.bpo-27827.TMWh1i.rst deleted file mode 100644 index 1b8cc04533ed8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-12-21-25-35.bpo-27827.TMWh1i.rst +++ /dev/null @@ -1,2 +0,0 @@ -:meth:`pathlib.PureWindowsPath.is_reserved` now identifies a greater range of -reserved filenames, including those with trailing spaces or colons. diff --git a/Misc/NEWS.d/next/Library/2021-06-12-22-58-20.bpo-44389.WTRnoC.rst b/Misc/NEWS.d/next/Library/2021-06-12-22-58-20.bpo-44389.WTRnoC.rst deleted file mode 100644 index e7e3b87489900..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-12-22-58-20.bpo-44389.WTRnoC.rst +++ /dev/null @@ -1 +0,0 @@ -Fix deprecation of :data:`ssl.OP_NO_TLSv1_3` diff --git a/Misc/NEWS.d/next/Library/2021-06-13-00-16-56.bpo-37880.5bTrkw.rst b/Misc/NEWS.d/next/Library/2021-06-13-00-16-56.bpo-37880.5bTrkw.rst deleted file mode 100644 index 42821572aa67d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-13-00-16-56.bpo-37880.5bTrkw.rst +++ /dev/null @@ -1,3 +0,0 @@ -argparse actions store_const and append_const each receive a default value -of None when the ``const`` kwarg is not provided. Previously, this raised a -:exc:`TypeError`. diff --git a/Misc/NEWS.d/next/Library/2021-06-14-14-19-11.bpo-38291.ee4cSX.rst b/Misc/NEWS.d/next/Library/2021-06-14-14-19-11.bpo-38291.ee4cSX.rst deleted file mode 100644 index 078d78d1778b1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-14-14-19-11.bpo-38291.ee4cSX.rst +++ /dev/null @@ -1 +0,0 @@ -Importing typing.io or typing.re now prints a ``DeprecationWarning``. diff --git a/Misc/NEWS.d/next/Library/2021-06-14-23-28-17.bpo-44422.BlWOgv.rst b/Misc/NEWS.d/next/Library/2021-06-14-23-28-17.bpo-44422.BlWOgv.rst deleted file mode 100644 index 09bace01fc779..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-14-23-28-17.bpo-44422.BlWOgv.rst +++ /dev/null @@ -1,3 +0,0 @@ -The :func:`threading.enumerate` function now uses a reentrant lock to -prevent a hang on reentrant call. -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-06-15-13-51-25.bpo-42972.UnyYo1.rst b/Misc/NEWS.d/next/Library/2021-06-15-13-51-25.bpo-42972.UnyYo1.rst deleted file mode 100644 index fbcc12c9f90a2..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-15-13-51-25.bpo-42972.UnyYo1.rst +++ /dev/null @@ -1,2 +0,0 @@ -The _thread.RLock type now fully implement the GC protocol: add a traverse -function and the :const:`Py_TPFLAGS_HAVE_GC` flag. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-06-16-16-52-14.bpo-44434.SQS4Pg.rst b/Misc/NEWS.d/next/Library/2021-06-16-16-52-14.bpo-44434.SQS4Pg.rst deleted file mode 100644 index 37b5b57ce6569..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-16-16-52-14.bpo-44434.SQS4Pg.rst +++ /dev/null @@ -1,4 +0,0 @@ -_thread.start_new_thread() no longer calls PyThread_exit_thread() explicitly -at the thread exit, the call was redundant. On Linux with the glibc, -pthread_exit() aborts the whole process if dlopen() fails to open -libgcc_s.so file (ex: EMFILE error). Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-06-17-15-01-51.bpo-44439.1S7QhT.rst b/Misc/NEWS.d/next/Library/2021-06-17-15-01-51.bpo-44439.1S7QhT.rst deleted file mode 100644 index 27396683700a8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-17-15-01-51.bpo-44439.1S7QhT.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix in :meth:`bz2.BZ2File.write` / :meth:`lzma.LZMAFile.write` methods, when -the input data is an object that supports the buffer protocol, the file length -may be wrong. diff --git a/Misc/NEWS.d/next/Library/2021-06-17-22-39-34.bpo-44446.qwdRic.rst b/Misc/NEWS.d/next/Library/2021-06-17-22-39-34.bpo-44446.qwdRic.rst deleted file mode 100644 index 6d9758f42dd04..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-17-22-39-34.bpo-44446.qwdRic.rst +++ /dev/null @@ -1 +0,0 @@ -Take into account that ``lineno`` might be ``None`` in :class:`traceback.FrameSummary`. diff --git a/Misc/NEWS.d/next/Library/2021-06-19-21-52-27.bpo-44464.U2oa-a.rst b/Misc/NEWS.d/next/Library/2021-06-19-21-52-27.bpo-44464.U2oa-a.rst deleted file mode 100644 index 6b1c10783d569..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-19-21-52-27.bpo-44464.U2oa-a.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove exception for flake8 in deprecated importlib.metadata interfaces. -Sync with importlib_metadata 4.6. diff --git a/Misc/NEWS.d/next/Library/2021-06-20-07-14-46.bpo-44458.myqCQ0.rst b/Misc/NEWS.d/next/Library/2021-06-20-07-14-46.bpo-44458.myqCQ0.rst deleted file mode 100644 index f15104b75e31c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-20-07-14-46.bpo-44458.myqCQ0.rst +++ /dev/null @@ -1 +0,0 @@ -``BUFFER_BLOCK_SIZE`` is now declared static, to avoid linking collisions when bz2, lmza or zlib are statically linked. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-06-20-14-03-18.bpo-41546.lO1jXU.rst b/Misc/NEWS.d/next/Library/2021-06-20-14-03-18.bpo-41546.lO1jXU.rst deleted file mode 100644 index 050da761570d4..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-20-14-03-18.bpo-41546.lO1jXU.rst +++ /dev/null @@ -1 +0,0 @@ -Make :mod:`pprint` (like the builtin ``print``) not attempt to write to ``stdout`` when it is ``None``. diff --git a/Misc/NEWS.d/next/Library/2021-06-20-19-01-11.bpo-44404.McfrYB.rst b/Misc/NEWS.d/next/Library/2021-06-20-19-01-11.bpo-44404.McfrYB.rst deleted file mode 100644 index ff6ca1bfa7242..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-20-19-01-11.bpo-44404.McfrYB.rst +++ /dev/null @@ -1 +0,0 @@ -:mod:`tkinter`'s ``after()`` method now supports callables without the ``__name__`` attribute. diff --git a/Misc/NEWS.d/next/Library/2021-06-21-10-46-58.bpo-44471.2QjXv_.rst b/Misc/NEWS.d/next/Library/2021-06-21-10-46-58.bpo-44471.2QjXv_.rst deleted file mode 100644 index 0675ef3262a09..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-21-10-46-58.bpo-44471.2QjXv_.rst +++ /dev/null @@ -1,5 +0,0 @@ -A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in -:meth:`contextlib.ExitStack.enter_context` and -:meth:`contextlib.AsyncExitStack.enter_async_context` for objects which do -not support the :term:`context manager` or :term:`asynchronous context -manager` protocols correspondingly. diff --git a/Misc/NEWS.d/next/Library/2021-06-21-12-43-04.bpo-44466.NSm6mv.rst b/Misc/NEWS.d/next/Library/2021-06-21-12-43-04.bpo-44466.NSm6mv.rst deleted file mode 100644 index 69de3edb5a7f9..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-21-12-43-04.bpo-44466.NSm6mv.rst +++ /dev/null @@ -1,2 +0,0 @@ -The :mod:`faulthandler` module now detects if a fatal error occurs during a -garbage collector collection. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-06-22-08-43-04.bpo-44482.U9GznK.rst b/Misc/NEWS.d/next/Library/2021-06-22-08-43-04.bpo-44482.U9GznK.rst deleted file mode 100644 index d05fe908e3eba..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-22-08-43-04.bpo-44482.U9GznK.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix very unlikely resource leak in :mod:`glob` in alternate Python -implementations. diff --git a/Misc/NEWS.d/next/Library/2021-06-22-16-45-48.bpo-43977.bamAGF.rst b/Misc/NEWS.d/next/Library/2021-06-22-16-45-48.bpo-43977.bamAGF.rst deleted file mode 100644 index 5f8cb7b7ea729..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-22-16-45-48.bpo-43977.bamAGF.rst +++ /dev/null @@ -1,3 +0,0 @@ -Set the proper :const:`Py_TPFLAGS_MAPPING` and :const:`Py_TPFLAGS_SEQUENCE` -flags for subclasses created before a parent has been registered as a -:class:`collections.abc.Mapping` or :class:`collections.abc.Sequence`. diff --git a/Misc/NEWS.d/next/Library/2021-06-23-01-33-01.bpo-44491.tiOlr5.rst b/Misc/NEWS.d/next/Library/2021-06-23-01-33-01.bpo-44491.tiOlr5.rst deleted file mode 100644 index ebe54484187ab..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-23-01-33-01.bpo-44491.tiOlr5.rst +++ /dev/null @@ -1,3 +0,0 @@ -Allow clearing the :mod:`sqlite3` authorizer callback by passing -:const:`None` to :meth:`~sqlite3.Connection.set_authorizer`. Patch by -Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-06-23-19-02-00.bpo-44468.-klV5-.rst b/Misc/NEWS.d/next/Library/2021-06-23-19-02-00.bpo-44468.-klV5-.rst deleted file mode 100644 index 78251c7d55068..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-23-19-02-00.bpo-44468.-klV5-.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`typing.get_type_hints` now finds annotations in classes and base classes -with unexpected ``__module__``. Previously, it skipped those MRO elements. diff --git a/Misc/NEWS.d/next/Library/2021-06-24-19-16-20.bpo-42892.qvRNhI.rst b/Misc/NEWS.d/next/Library/2021-06-24-19-16-20.bpo-42892.qvRNhI.rst deleted file mode 100644 index 3c70b0534ecab..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-24-19-16-20.bpo-42892.qvRNhI.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed an exception thrown while parsing a malformed multipart email by :class:`email.message.EmailMessage`. diff --git a/Misc/NEWS.d/next/Library/2021-06-26-12-27-14.bpo-44516.BVyX_y.rst b/Misc/NEWS.d/next/Library/2021-06-26-12-27-14.bpo-44516.BVyX_y.rst deleted file mode 100644 index a9822881135ea..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-26-12-27-14.bpo-44516.BVyX_y.rst +++ /dev/null @@ -1 +0,0 @@ -Update vendored pip to 21.1.3 diff --git a/Misc/NEWS.d/next/Library/2021-06-29-07-27-08.bpo-43625.ZlAxhp.rst b/Misc/NEWS.d/next/Library/2021-06-29-07-27-08.bpo-43625.ZlAxhp.rst deleted file mode 100644 index a21975b948ef9..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-29-07-27-08.bpo-43625.ZlAxhp.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix a bug in the detection of CSV file headers by -:meth:`csv.Sniffer.has_header` and improve documentation of same. diff --git a/Misc/NEWS.d/next/Library/2021-06-29-21-17-17.bpo-44461.acqRnV.rst b/Misc/NEWS.d/next/Library/2021-06-29-21-17-17.bpo-44461.acqRnV.rst deleted file mode 100644 index 02e25e928b9cf..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-29-21-17-17.bpo-44461.acqRnV.rst +++ /dev/null @@ -1 +0,0 @@ -Fix bug with :mod:`pdb`'s handling of import error due to a package which does not have a ``__main__`` module \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-06-30-11-34-35.bpo-44539.nP0Xi4.rst b/Misc/NEWS.d/next/Library/2021-06-30-11-34-35.bpo-44539.nP0Xi4.rst deleted file mode 100644 index f5e831afce835..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-30-11-34-35.bpo-44539.nP0Xi4.rst +++ /dev/null @@ -1 +0,0 @@ -Added support for recognizing JPEG files without JFIF or Exif markers. diff --git a/Misc/NEWS.d/next/Library/2021-06-30-13-29-49.bpo-34798.t7FCa0.rst b/Misc/NEWS.d/next/Library/2021-06-30-13-29-49.bpo-34798.t7FCa0.rst deleted file mode 100644 index ab9fd8e33799d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-06-30-13-29-49.bpo-34798.t7FCa0.rst +++ /dev/null @@ -1 +0,0 @@ -Break up paragraph about :class:`pprint.PrettyPrinter` construction parameters to make it easier to read. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-07-02-18-17-56.bpo-44554.aBUmJo.rst b/Misc/NEWS.d/next/Library/2021-07-02-18-17-56.bpo-44554.aBUmJo.rst deleted file mode 100644 index 2c225b8083951..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-02-18-17-56.bpo-44554.aBUmJo.rst +++ /dev/null @@ -1 +0,0 @@ -Refactor argument processing in :func:`pdb.main` to simplify detection of errors in input loading and clarify behavior around module or script invocation. diff --git a/Misc/NEWS.d/next/Library/2021-07-04-11-33-34.bpo-41249.sHdwBE.rst b/Misc/NEWS.d/next/Library/2021-07-04-11-33-34.bpo-41249.sHdwBE.rst deleted file mode 100644 index 06dae4a6e9356..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-04-11-33-34.bpo-41249.sHdwBE.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixes ``TypedDict`` to work with ``typing.get_type_hints()`` and postponed evaluation of -annotations across modules. diff --git a/Misc/NEWS.d/next/Library/2021-07-04-21-16-53.bpo-44558.cm7Slv.rst b/Misc/NEWS.d/next/Library/2021-07-04-21-16-53.bpo-44558.cm7Slv.rst deleted file mode 100644 index 647a70490d1ba..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-04-21-16-53.bpo-44558.cm7Slv.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make the implementation consistency of :func:`~operator.indexOf` between -C and Python versions. Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-07-05-18-13-25.bpo-44566.o51Bd1.rst b/Misc/NEWS.d/next/Library/2021-07-05-18-13-25.bpo-44566.o51Bd1.rst deleted file mode 100644 index 3b00a1b715fee..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-05-18-13-25.bpo-44566.o51Bd1.rst +++ /dev/null @@ -1 +0,0 @@ -handle StopIteration subclass raised from @contextlib.contextmanager generator \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-07-08-12-22-54.bpo-44569.KZ02v9.rst b/Misc/NEWS.d/next/Library/2021-07-08-12-22-54.bpo-44569.KZ02v9.rst deleted file mode 100644 index 5f693b290dfb8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-08-12-22-54.bpo-44569.KZ02v9.rst +++ /dev/null @@ -1,3 +0,0 @@ -Added the :func:`StackSummary.format_frame` function in :mod:`traceback`. -This allows users to customize the way individual lines are formatted in -tracebacks without re-implementing logic to handle recursive tracebacks. diff --git a/Misc/NEWS.d/next/Library/2021-07-09-07-14-37.bpo-41928.Q1jMrr.rst b/Misc/NEWS.d/next/Library/2021-07-09-07-14-37.bpo-41928.Q1jMrr.rst deleted file mode 100644 index e6bd758980b00..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-09-07-14-37.bpo-41928.Q1jMrr.rst +++ /dev/null @@ -1,4 +0,0 @@ -Update :func:`shutil.copyfile` to raise :exc:`FileNotFoundError` instead of -confusing :exc:`IsADirectoryError` when a path ending with a -:const:`os.path.sep` does not exist; :func:`shutil.copy` and -:func:`shutil.copy2` are also affected. diff --git a/Misc/NEWS.d/next/Library/2021-07-10-19-55-13.bpo-42799.ad4tq8.rst b/Misc/NEWS.d/next/Library/2021-07-10-19-55-13.bpo-42799.ad4tq8.rst deleted file mode 100644 index 8a25800611a5a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-10-19-55-13.bpo-42799.ad4tq8.rst +++ /dev/null @@ -1,4 +0,0 @@ -In :mod:`fnmatch`, the cache size for compiled regex patterns -(:func:`functools.lru_cache`) was bumped up from 256 to 32768, affecting -functions: :func:`fnmatch.fnmatch`, :func:`fnmatch.fnmatchcase`, -:func:`fnmatch.filter`. diff --git a/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst b/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst deleted file mode 100644 index a2bfd8ff5b51b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-12-10-32-48.bpo-44594.eEa5zi.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix an edge case of :class:`ExitStack` and :class:`AsyncExitStack` exception -chaining. They will now match ``with`` block behavior when ``__context__`` is -explicitly set to ``None`` when the exception is in flight. diff --git a/Misc/NEWS.d/next/Library/2021-07-13-09-01-33.bpo-44608.R3IcM1.rst b/Misc/NEWS.d/next/Library/2021-07-13-09-01-33.bpo-44608.R3IcM1.rst deleted file mode 100644 index e0cf948f3cba6..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-13-09-01-33.bpo-44608.R3IcM1.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix memory leak in :func:`_tkinter._flatten` if it is called with a sequence -or set, but not list or tuple. diff --git a/Misc/NEWS.d/next/Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst b/Misc/NEWS.d/next/Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst deleted file mode 100644 index f7171c3c84c5e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-15-16-51-32.bpo-44648.2o49TB.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed wrong error being thrown by :func:`inspect.getsource` when examining a -class in the interactive session. Instead of :exc:`TypeError`, it should be -:exc:`OSError` with appropriate error message. diff --git a/Misc/NEWS.d/next/Library/2021-07-16-08-57-27.bpo-44638.EwYKne.rst b/Misc/NEWS.d/next/Library/2021-07-16-08-57-27.bpo-44638.EwYKne.rst deleted file mode 100644 index eeaa91c16c4cf..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-16-08-57-27.bpo-44638.EwYKne.rst +++ /dev/null @@ -1 +0,0 @@ -Add a reference to the zipp project and hint as to how to use it. diff --git a/Misc/NEWS.d/next/Library/2021-07-16-13-40-31.bpo-40897.aveAre.rst b/Misc/NEWS.d/next/Library/2021-07-16-13-40-31.bpo-40897.aveAre.rst deleted file mode 100644 index 04f1465f0ac67..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-16-13-40-31.bpo-40897.aveAre.rst +++ /dev/null @@ -1,2 +0,0 @@ -Give priority to using the current class constructor in -:func:`inspect.signature`. Patch by Weipeng Hong. diff --git a/Misc/NEWS.d/next/Library/2021-07-19-14-04-42.bpo-44524.Nbf2JC.rst b/Misc/NEWS.d/next/Library/2021-07-19-14-04-42.bpo-44524.Nbf2JC.rst deleted file mode 100644 index 0acdc7dff029f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-19-14-04-42.bpo-44524.Nbf2JC.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add missing ``__name__`` and ``__qualname__`` attributes to ``typing`` module -classes. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-07-19-18-45-00.bpo-44678.YMEAu0.rst b/Misc/NEWS.d/next/Library/2021-07-19-18-45-00.bpo-44678.YMEAu0.rst deleted file mode 100644 index 991b1579b2e2b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-19-18-45-00.bpo-44678.YMEAu0.rst +++ /dev/null @@ -1 +0,0 @@ -Added a separate error message for discontinuous padding in *binascii.a2b_base64* strict mode. diff --git a/Misc/NEWS.d/next/Library/2021-07-19-22-43-15.bpo-44353.HF81_Q.rst b/Misc/NEWS.d/next/Library/2021-07-19-22-43-15.bpo-44353.HF81_Q.rst deleted file mode 100644 index 8a1e0f77b3177..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-19-22-43-15.bpo-44353.HF81_Q.rst +++ /dev/null @@ -1,2 +0,0 @@ -Refactor ``typing.NewType`` from function into callable class. Patch -provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-07-20-00-11-47.bpo-44682.3m2qVV.rst b/Misc/NEWS.d/next/Library/2021-07-20-00-11-47.bpo-44682.3m2qVV.rst deleted file mode 100644 index 308053a62c389..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-00-11-47.bpo-44682.3m2qVV.rst +++ /dev/null @@ -1,2 +0,0 @@ -Change the :mod:`pdb` *commands* directive to disallow setting commands -for an invalid breakpoint and to display an appropriate error. diff --git a/Misc/NEWS.d/next/Library/2021-07-20-18-34-16.bpo-44353.ATuYq4.rst b/Misc/NEWS.d/next/Library/2021-07-20-18-34-16.bpo-44353.ATuYq4.rst deleted file mode 100644 index 7332770ac487e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-18-34-16.bpo-44353.ATuYq4.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make ``NewType.__call__`` faster by implementing it in C. -Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-07-20-19-35-49.bpo-44686.ucCGhu.rst b/Misc/NEWS.d/next/Library/2021-07-20-19-35-49.bpo-44686.ucCGhu.rst deleted file mode 100644 index d9c78020e4a9b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-19-35-49.bpo-44686.ucCGhu.rst +++ /dev/null @@ -1 +0,0 @@ -Replace ``unittest.mock._importer`` with ``pkgutil.resolve_name``. diff --git a/Misc/NEWS.d/next/Library/2021-07-20-21-51-35.bpo-42854.ThuDMI.rst b/Misc/NEWS.d/next/Library/2021-07-20-21-51-35.bpo-42854.ThuDMI.rst deleted file mode 100644 index 33cbb63a5e14b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-21-51-35.bpo-42854.ThuDMI.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed a bug in the :mod:`_ssl` module that was throwing :exc:`OverflowError` -when using :meth:`_ssl._SSLSocket.write` and :meth:`_ssl._SSLSocket.read` -for a big value of the ``len`` parameter. Patch by Pablo Galindo diff --git a/Misc/NEWS.d/next/Library/2021-07-20-22-03-24.bpo-44690.tV7Zjg.rst b/Misc/NEWS.d/next/Library/2021-07-20-22-03-24.bpo-44690.tV7Zjg.rst deleted file mode 100644 index 1d1184805471d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-22-03-24.bpo-44690.tV7Zjg.rst +++ /dev/null @@ -1 +0,0 @@ -Adopt *binacii.a2b_base64*'s strict mode in *base64.b64decode*. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-07-20-23-28-26.bpo-44688.buFgz3.rst b/Misc/NEWS.d/next/Library/2021-07-20-23-28-26.bpo-44688.buFgz3.rst deleted file mode 100644 index 15f6a521f2d4f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-20-23-28-26.bpo-44688.buFgz3.rst +++ /dev/null @@ -1,2 +0,0 @@ -:meth:`sqlite3.Connection.create_collation` now accepts non-ASCII collation -names. Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-07-21-10-43-22.bpo-44666.CEThkv.rst b/Misc/NEWS.d/next/Library/2021-07-21-10-43-22.bpo-44666.CEThkv.rst deleted file mode 100644 index ab2ef22d0c455..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-21-10-43-22.bpo-44666.CEThkv.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed issue in :func:`compileall.compile_file` when ``sys.stdout`` is redirected. -Patch by Stefan H?lzl. diff --git a/Misc/NEWS.d/next/Library/2021-07-21-23-16-30.bpo-44704.iqHLxQ.rst b/Misc/NEWS.d/next/Library/2021-07-21-23-16-30.bpo-44704.iqHLxQ.rst deleted file mode 100644 index 586661876dedb..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-21-23-16-30.bpo-44704.iqHLxQ.rst +++ /dev/null @@ -1 +0,0 @@ -The implementation of ``collections.abc.Set._hash()`` now matches that of ``frozenset.__hash__()``. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-07-24-02-17-59.bpo-44720.shU5Qm.rst b/Misc/NEWS.d/next/Library/2021-07-24-02-17-59.bpo-44720.shU5Qm.rst deleted file mode 100644 index 83694f3988ae9..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-24-02-17-59.bpo-44720.shU5Qm.rst +++ /dev/null @@ -1 +0,0 @@ -``weakref.proxy`` objects referencing non-iterators now raise ``TypeError`` rather than dereferencing the null ``tp_iternext`` slot and crashing. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-07-25-08-17-55.bpo-42378.WIhUZK.rst b/Misc/NEWS.d/next/Library/2021-07-25-08-17-55.bpo-42378.WIhUZK.rst deleted file mode 100644 index 90c3961dc87d8..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-25-08-17-55.bpo-42378.WIhUZK.rst +++ /dev/null @@ -1,4 +0,0 @@ -Fixes the issue with log file being overwritten when -:class:`logging.FileHandler` is used in :mod:`atexit` with *filemode* set to -``'w'``. Note this will cause the message in *atexit* not being logged if -the log stream is already closed due to shutdown of logging. diff --git a/Misc/NEWS.d/next/Library/2021-07-27-12-06-19.bpo-44747.epUzZz.rst b/Misc/NEWS.d/next/Library/2021-07-27-12-06-19.bpo-44747.epUzZz.rst deleted file mode 100644 index e63d77f76de92..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-27-12-06-19.bpo-44747.epUzZz.rst +++ /dev/null @@ -1,2 +0,0 @@ -Refactor usage of ``sys._getframe`` in ``typing`` module. Patch provided by -Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-07-27-22-11-29.bpo-44752._bvbrZ.rst b/Misc/NEWS.d/next/Library/2021-07-27-22-11-29.bpo-44752._bvbrZ.rst deleted file mode 100644 index 0d8a2cd6a5e0d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-27-22-11-29.bpo-44752._bvbrZ.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`rcompleter` does not call :func:`getattr` on :class:`property` objects -to avoid the side-effect of evaluating the corresponding method. diff --git a/Misc/NEWS.d/next/Library/2021-07-28-22-53-18.bpo-44771.BvLdnU.rst b/Misc/NEWS.d/next/Library/2021-07-28-22-53-18.bpo-44771.BvLdnU.rst deleted file mode 100644 index 0d47a55a7d74f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-28-22-53-18.bpo-44771.BvLdnU.rst +++ /dev/null @@ -1,5 +0,0 @@ -Added ``importlib.simple`` module implementing adapters from a low-level -resources reader interface to a ``TraversableResources`` interface. Legacy -API (``path``, ``contents``, ...) is now supported entirely by the -``.files()`` API with a compatibility shim supplied for resource loaders -without that functionality. Feature parity with ``importlib_resources`` 5.2. diff --git a/Misc/NEWS.d/next/Library/2021-07-30-23-27-30.bpo-44667.tu0Xrv.rst b/Misc/NEWS.d/next/Library/2021-07-30-23-27-30.bpo-44667.tu0Xrv.rst deleted file mode 100644 index 5b7e20e0afdf5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-30-23-27-30.bpo-44667.tu0Xrv.rst +++ /dev/null @@ -1,4 +0,0 @@ -The :func:`tokenize.tokenize` doesn't incorrectly generate a ``NEWLINE`` -token if the source doesn't end with a new line character but the last line -is a comment, as the function is already generating a ``NL`` token. Patch by -Pablo Galindo diff --git a/Misc/NEWS.d/next/Library/2021-07-31-08-45-31.bpo-44784.fIMIDS.rst b/Misc/NEWS.d/next/Library/2021-07-31-08-45-31.bpo-44784.fIMIDS.rst deleted file mode 100644 index 6ad10ef3f5980..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-31-08-45-31.bpo-44784.fIMIDS.rst +++ /dev/null @@ -1,2 +0,0 @@ -In importlib.metadata tests, override warnings behavior under expected -DeprecationWarnings (importlib_metadata 4.6.3). diff --git a/Misc/NEWS.d/next/Library/2021-07-31-20-28-20.bpo-44793.woaQSg.rst b/Misc/NEWS.d/next/Library/2021-07-31-20-28-20.bpo-44793.woaQSg.rst deleted file mode 100644 index 1d94d67615a47..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-07-31-20-28-20.bpo-44793.woaQSg.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix checking the number of arguments when subscribe a generic type with -``ParamSpec`` parameter. diff --git a/Misc/NEWS.d/next/Library/2021-08-01-19-49-09.bpo-27275.QsvE0k.rst b/Misc/NEWS.d/next/Library/2021-08-01-19-49-09.bpo-27275.QsvE0k.rst deleted file mode 100644 index 1f5afaf4d108e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-01-19-49-09.bpo-27275.QsvE0k.rst +++ /dev/null @@ -1,3 +0,0 @@ -:meth:`collections.OrderedDict.popitem` and :meth:`collections.OrderedDict.pop` -no longer call ``__getitem__`` and ``__delitem__`` methods of the OrderedDict -subclasses. diff --git a/Misc/NEWS.d/next/Library/2021-08-02-14-37-32.bpo-44806.wOW_Qn.rst b/Misc/NEWS.d/next/Library/2021-08-02-14-37-32.bpo-44806.wOW_Qn.rst deleted file mode 100644 index 6d818c3fc5798..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-02-14-37-32.bpo-44806.wOW_Qn.rst +++ /dev/null @@ -1,2 +0,0 @@ -Non-protocol subclasses of :class:`typing.Protocol` ignore now the -``__init__`` method inherited from protocol base classes. diff --git a/Misc/NEWS.d/next/Library/2021-08-03-20-37-45.bpo-44801.i49Aug.rst b/Misc/NEWS.d/next/Library/2021-08-03-20-37-45.bpo-44801.i49Aug.rst deleted file mode 100644 index 05e372a5fabb0..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-03-20-37-45.bpo-44801.i49Aug.rst +++ /dev/null @@ -1,3 +0,0 @@ -Ensure that the :class:`~typing.ParamSpec` variable in Callable -can only be substituted with a parameters expression (a list of types, -an ellipsis, ParamSpec or Concatenate). diff --git a/Misc/NEWS.d/next/Library/2021-08-04-12-29-00.bpo-44822.zePNXA.rst b/Misc/NEWS.d/next/Library/2021-08-04-12-29-00.bpo-44822.zePNXA.rst deleted file mode 100644 index d078142886d2e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-04-12-29-00.bpo-44822.zePNXA.rst +++ /dev/null @@ -1,3 +0,0 @@ -:mod:`sqlite3` user-defined functions and aggregators returning -:class:`strings ` with embedded NUL characters are no longer -truncated. Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-08-05-14-59-39.bpo-44839.MURNL9.rst b/Misc/NEWS.d/next/Library/2021-08-05-14-59-39.bpo-44839.MURNL9.rst deleted file mode 100644 index 62ad62c5d48d5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-05-14-59-39.bpo-44839.MURNL9.rst +++ /dev/null @@ -1,4 +0,0 @@ -:class:`MemoryError` raised in user-defined functions will now produce a -``MemoryError`` in :mod:`sqlite3`. :class:`OverflowError` will now be converted -to :class:`~sqlite3.DataError`. Previously -:class:`~sqlite3.OperationalError` was produced in these cases. diff --git a/Misc/NEWS.d/next/Library/2021-08-05-18-20-17.bpo-44524.9T1tfe.rst b/Misc/NEWS.d/next/Library/2021-08-05-18-20-17.bpo-44524.9T1tfe.rst deleted file mode 100644 index 0c9e82050883d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-05-18-20-17.bpo-44524.9T1tfe.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixed an issue wherein the ``__name__`` and ``__qualname__`` attributes of -subscribed specialforms could be ``None``. diff --git a/Misc/NEWS.d/next/Library/2021-08-06-09-43-50.bpo-44605.q4YSBZ.rst b/Misc/NEWS.d/next/Library/2021-08-06-09-43-50.bpo-44605.q4YSBZ.rst deleted file mode 100644 index 93783923e15b3..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-06-09-43-50.bpo-44605.q4YSBZ.rst +++ /dev/null @@ -1 +0,0 @@ -The @functools.total_ordering() decorator now works with metaclasses. diff --git a/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst b/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst deleted file mode 100644 index b1f225485ddef..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst +++ /dev/null @@ -1,4 +0,0 @@ -Fix the :func:`os.set_inheritable` function on FreeBSD 14 for file descriptor -opened with the :data:`~os.O_PATH` flag: ignore the :data:`~errno.EBADF` -error on ``ioctl()``, fallback on the ``fcntl()`` implementation. Patch by -Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-08-06-19-15-52.bpo-44581.oFDBTB.rst b/Misc/NEWS.d/next/Library/2021-08-06-19-15-52.bpo-44581.oFDBTB.rst deleted file mode 100644 index 99f08065b9d2f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-06-19-15-52.bpo-44581.oFDBTB.rst +++ /dev/null @@ -1 +0,0 @@ -Upgrade bundled pip to 21.2.3 and setuptools to 57.4.0 diff --git a/Misc/NEWS.d/next/Library/2021-08-07-17-28-56.bpo-44859.CCopjk.rst b/Misc/NEWS.d/next/Library/2021-08-07-17-28-56.bpo-44859.CCopjk.rst deleted file mode 100644 index ec9f774d66b8c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-07-17-28-56.bpo-44859.CCopjk.rst +++ /dev/null @@ -1,8 +0,0 @@ -Improve error handling in :mod:`sqlite3` and raise more accurate exceptions. - -* :exc:`MemoryError` is now raised instead of :exc:`sqlite3.Warning` when memory is not enough for encoding a statement to UTF-8 in ``Connection.__call__()`` and ``Cursor.execute()``. -* :exc:`UnicodEncodeError` is now raised instead of :exc:`sqlite3.Warning` when the statement contains surrogate characters in ``Connection.__call__()`` and ``Cursor.execute()``. -* :exc:`TypeError` is now raised instead of :exc:`ValueError` for non-string script argument in ``Cursor.executescript()``. -* :exc:`ValueError` is now raised for script containing the null character instead of truncating it in ``Cursor.executescript()``. -* Correctly handle exceptions raised when getting boolean value of the result of the progress handler. -* Add many tests covering different corner cases. diff --git a/Misc/NEWS.d/next/Library/2021-08-07-22-51-32.bpo-44860.PTvRrU.rst b/Misc/NEWS.d/next/Library/2021-08-07-22-51-32.bpo-44860.PTvRrU.rst deleted file mode 100644 index 70e0be0dab40c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-07-22-51-32.bpo-44860.PTvRrU.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix the ``posix_user`` scheme in :mod:`sysconfig` to not depend on -:data:`sys.platlibdir`. diff --git a/Misc/NEWS.d/next/Library/2021-08-09-13-17-10.bpo-38956.owWLNv.rst b/Misc/NEWS.d/next/Library/2021-08-09-13-17-10.bpo-38956.owWLNv.rst deleted file mode 100644 index 3f57c0ea5d5a3..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-09-13-17-10.bpo-38956.owWLNv.rst +++ /dev/null @@ -1 +0,0 @@ -:class:`argparse.BooleanOptionalAction`'s default value is no longer printed twice when used with :class:`argparse.ArgumentDefaultsHelpFormatter`. diff --git a/Misc/NEWS.d/next/Library/2021-08-10-16-57-10.bpo-44524.dk9QX4.rst b/Misc/NEWS.d/next/Library/2021-08-10-16-57-10.bpo-44524.dk9QX4.rst deleted file mode 100644 index bc3659fca5209..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-10-16-57-10.bpo-44524.dk9QX4.rst +++ /dev/null @@ -1,2 +0,0 @@ -Make exception message more useful when subclass from typing special form -alias. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-08-12-16-22-16.bpo-41322.utscTd.rst b/Misc/NEWS.d/next/Library/2021-08-12-16-22-16.bpo-41322.utscTd.rst deleted file mode 100644 index e16efd2c7bd55..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-12-16-22-16.bpo-41322.utscTd.rst +++ /dev/null @@ -1,3 +0,0 @@ -Added ``DeprecationWarning`` for tests and async tests that return a -value!=None (as this may indicate an improperly written test, for example a -test written as a generator function). diff --git a/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst b/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst deleted file mode 100644 index f8aed69a40a3b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-14-00-55-16.bpo-44911.uk3hYk.rst +++ /dev/null @@ -1 +0,0 @@ -:class:`~unittest.IsolatedAsyncioTestCase` will no longer throw an exception while cancelling leaked tasks. Patch by Bar Harel. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-08-17-16-01-44.bpo-44935.roUl0G.rst b/Misc/NEWS.d/next/Library/2021-08-17-16-01-44.bpo-44935.roUl0G.rst deleted file mode 100644 index 3d41c3be1403d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-17-16-01-44.bpo-44935.roUl0G.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`subprocess` on Solaris now also uses :func:`os.posix_spawn()` for -better performance. diff --git a/Misc/NEWS.d/next/Library/2021-08-18-10-36-14.bpo-39039.A63LYh.rst b/Misc/NEWS.d/next/Library/2021-08-18-10-36-14.bpo-39039.A63LYh.rst deleted file mode 100644 index 7250055c2a4a9..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-18-10-36-14.bpo-39039.A63LYh.rst +++ /dev/null @@ -1,2 +0,0 @@ -tarfile.open raises :exc:`~tarfile.ReadError` when a zlib error occurs -during file extraction. diff --git a/Misc/NEWS.d/next/Library/2021-08-19-15-03-54.bpo-44955.1mxFQS.rst b/Misc/NEWS.d/next/Library/2021-08-19-15-03-54.bpo-44955.1mxFQS.rst deleted file mode 100644 index 57d1da533cde0..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-19-15-03-54.bpo-44955.1mxFQS.rst +++ /dev/null @@ -1,5 +0,0 @@ -Method :meth:`~unittest.TestResult.stopTestRun` is now always called in pair -with method :meth:`~unittest.TestResult.startTestRun` for -:class:`~unittest.TestResult` objects implicitly created in -:meth:`~unittest.TestCase.run`. Previously it was not called for test -methods and classes decorated with a skipping decorator. diff --git a/Misc/NEWS.d/next/Library/2021-08-19-23-49-10.bpo-42255.ofe3ms.rst b/Misc/NEWS.d/next/Library/2021-08-19-23-49-10.bpo-42255.ofe3ms.rst deleted file mode 100644 index 84a02c4c3fb2b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-19-23-49-10.bpo-42255.ofe3ms.rst +++ /dev/null @@ -1,3 +0,0 @@ -:class:`webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. -It is untested and undocumented and also not used by webbrowser itself. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-08-22-13-25-17.bpo-44019.BN8HDy.rst b/Misc/NEWS.d/next/Library/2021-08-22-13-25-17.bpo-44019.BN8HDy.rst deleted file mode 100644 index 37556d76905d7..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-22-13-25-17.bpo-44019.BN8HDy.rst +++ /dev/null @@ -1,2 +0,0 @@ -A new function ``operator.call`` has been added, such that -``operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)``. diff --git a/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst b/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst deleted file mode 100644 index 81fdfeb629456..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-23-21-39-59.bpo-37596.ojRcwB.rst +++ /dev/null @@ -1,2 +0,0 @@ -Ensure that :class:`set` and :class:`frozenset` objects are always -:mod:`marshalled ` reproducibly. diff --git a/Misc/NEWS.d/next/Library/2021-08-25-10-28-49.bpo-43613.WkYmI0.rst b/Misc/NEWS.d/next/Library/2021-08-25-10-28-49.bpo-43613.WkYmI0.rst deleted file mode 100644 index d6af35c12b3c7..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-25-10-28-49.bpo-43613.WkYmI0.rst +++ /dev/null @@ -1,3 +0,0 @@ -Improve the speed of :func:`gzip.compress` and :func:`gzip.decompress` by -compressing and decompressing at once in memory instead of in a streamed -fashion. diff --git a/Misc/NEWS.d/next/Library/2021-08-25-20-18-31.bpo-39218.BlO6jW.rst b/Misc/NEWS.d/next/Library/2021-08-25-20-18-31.bpo-39218.BlO6jW.rst deleted file mode 100644 index f45dbfe463c87..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-25-20-18-31.bpo-39218.BlO6jW.rst +++ /dev/null @@ -1 +0,0 @@ -Improve accuracy of variance calculations by using ``x*x`` instead of ``x**2``. diff --git a/Misc/NEWS.d/next/Library/2021-08-26-09-54-14.bpo-45010.Cn23bQ.rst b/Misc/NEWS.d/next/Library/2021-08-26-09-54-14.bpo-45010.Cn23bQ.rst deleted file mode 100644 index bdf1bfe1ffe06..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-26-09-54-14.bpo-45010.Cn23bQ.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove support of special method ``__div__`` in :mod:`unittest.mock`. It is -not used in Python 3. diff --git a/Misc/NEWS.d/next/Library/2021-08-26-16-25-48.bpo-45001.tn_dKp.rst b/Misc/NEWS.d/next/Library/2021-08-26-16-25-48.bpo-45001.tn_dKp.rst deleted file mode 100644 index 55cc409d0da30..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-26-16-25-48.bpo-45001.tn_dKp.rst +++ /dev/null @@ -1,2 +0,0 @@ -Made email date parsing more robust against malformed input, namely a -whitespace-only ``Date:`` header. Patch by Wouter Bolsterlee. diff --git a/Misc/NEWS.d/next/Library/2021-08-27-19-01-23.bpo-45030.tAmBbY.rst b/Misc/NEWS.d/next/Library/2021-08-27-19-01-23.bpo-45030.tAmBbY.rst deleted file mode 100644 index dec8c88b15588..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-27-19-01-23.bpo-45030.tAmBbY.rst +++ /dev/null @@ -1 +0,0 @@ -Fix integer overflow in pickling and copying the range iterator. diff --git a/Misc/NEWS.d/next/Library/2021-08-27-23-40-51.bpo-43913.Uo1Gt5.rst b/Misc/NEWS.d/next/Library/2021-08-27-23-40-51.bpo-43913.Uo1Gt5.rst deleted file mode 100644 index cf3d5ee0e456f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-27-23-40-51.bpo-43913.Uo1Gt5.rst +++ /dev/null @@ -1,8 +0,0 @@ -Fix bugs in cleaning up classes and modules in :mod:`unittest`: - -* Functions registered with :func:`~unittest.addModuleCleanup` were not called unless the user defines ``tearDownModule()`` in their test module. -* Functions registered with :meth:`~unittest.TestCase.addClassCleanup` were not called if ``tearDownClass`` is set to ``None``. -* Buffering in :class:`~unittest.TestResult` did not work with functions registered with ``addClassCleanup()`` and ``addModuleCleanup()``. -* Errors in functions registered with ``addClassCleanup()`` and ``addModuleCleanup()`` were not handled correctly in buffered and debug modes. -* Errors in ``setUpModule()`` and functions registered with ``addModuleCleanup()`` were reported in wrong order. -* And several lesser bugs. diff --git a/Misc/NEWS.d/next/Library/2021-08-28-13-00-12.bpo-45021.rReeaj.rst b/Misc/NEWS.d/next/Library/2021-08-28-13-00-12.bpo-45021.rReeaj.rst deleted file mode 100644 index 54fd9109a9ae5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-28-13-00-12.bpo-45021.rReeaj.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a potential deadlock at shutdown of forked children when using :mod:`concurrent.futures` module \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-08-29-14-49-22.bpo-41620.WJ6PFL.rst b/Misc/NEWS.d/next/Library/2021-08-29-14-49-22.bpo-41620.WJ6PFL.rst deleted file mode 100644 index 7674d4c9532a2..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-29-14-49-22.bpo-41620.WJ6PFL.rst +++ /dev/null @@ -1,3 +0,0 @@ -:meth:`~unittest.TestCase.run` now always return a -:class:`~unittest.TestResult` instance. Previously it returned ``None`` if -the test class or method was decorated with a skipping decorator. diff --git a/Misc/NEWS.d/next/Library/2021-08-30-13-55-09.bpo-31299.9QzjZs.rst b/Misc/NEWS.d/next/Library/2021-08-30-13-55-09.bpo-31299.9QzjZs.rst deleted file mode 100644 index 1ffa0b15172ee..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-08-30-13-55-09.bpo-31299.9QzjZs.rst +++ /dev/null @@ -1 +0,0 @@ -Add option to completely drop frames from a traceback by returning ``None`` from a :meth:`~traceback.StackSummary.format_frame` override. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-09-01-15-27-00.bpo-45075.9xUpvt.rst b/Misc/NEWS.d/next/Library/2021-09-01-15-27-00.bpo-45075.9xUpvt.rst deleted file mode 100644 index 369b4506066e5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-01-15-27-00.bpo-45075.9xUpvt.rst +++ /dev/null @@ -1,5 +0,0 @@ -Rename :meth:`traceback.StackSummary.format_frame` to -:meth:`traceback.StackSummary.format_frame_summary`. This method was added -for 3.11 so it was not released yet. - -Updated code and docs to better distinguish frame and FrameSummary. diff --git a/Misc/NEWS.d/next/Library/2021-09-02-00-18-32.bpo-40360.9nmMtB.rst b/Misc/NEWS.d/next/Library/2021-09-02-00-18-32.bpo-40360.9nmMtB.rst deleted file mode 100644 index 4e9422dc06d7f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-02-00-18-32.bpo-40360.9nmMtB.rst +++ /dev/null @@ -1,3 +0,0 @@ -The :mod:`lib2to3` package is now deprecated and may not be able to parse -Python 3.10 or newer. See the :pep:`617` (New PEG parser for CPython). Patch -by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-09-02-00-47-14.bpo-45085.mMnaDv.rst b/Misc/NEWS.d/next/Library/2021-09-02-00-47-14.bpo-45085.mMnaDv.rst deleted file mode 100644 index 22eada24f0f5a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-02-00-47-14.bpo-45085.mMnaDv.rst +++ /dev/null @@ -1,10 +0,0 @@ -The ``binhex`` module, deprecated in Python 3.9, is now removed. The -following :mod:`binascii` functions, deprecated in Python 3.9, are now also -removed: - -* ``a2b_hqx()``, ``b2a_hqx()``; -* ``rlecode_hqx()``, ``rledecode_hqx()``. - -The :func:`binascii.crc_hqx` function remains available. - -Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-09-02-12-42-25.bpo-45081.tOjJ1k.rst b/Misc/NEWS.d/next/Library/2021-09-02-12-42-25.bpo-45081.tOjJ1k.rst deleted file mode 100644 index 86d7182003bb9..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-02-12-42-25.bpo-45081.tOjJ1k.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix issue when dataclasses that inherit from ``typing.Protocol`` subclasses -have wrong ``__init__``. Patch provided by Yurii Karabas. diff --git a/Misc/NEWS.d/next/Library/2021-09-05-13-15-08.bpo-25894.zjbi2f.rst b/Misc/NEWS.d/next/Library/2021-09-05-13-15-08.bpo-25894.zjbi2f.rst deleted file mode 100644 index b0a036fae6cfa..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-05-13-15-08.bpo-25894.zjbi2f.rst +++ /dev/null @@ -1,4 +0,0 @@ -:mod:`unittest` now always reports skipped and failed subtests separately: -separate characters in default mode and separate lines in verbose mode. Also -the test description is now output for errors in test method, class and -module cleanups. diff --git a/Misc/NEWS.d/next/Library/2021-09-05-20-33-25.bpo-45034.62NLD5.rst b/Misc/NEWS.d/next/Library/2021-09-05-20-33-25.bpo-45034.62NLD5.rst deleted file mode 100644 index 8d94821470a2a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-05-20-33-25.bpo-45034.62NLD5.rst +++ /dev/null @@ -1,2 +0,0 @@ -Changes how error is formatted for ``struct.pack`` with ``'H'`` and ``'h'`` modes and -too large / small numbers. Now it shows the actual numeric limits, while previously it was showing arithmetic expressions. diff --git a/Misc/NEWS.d/next/Library/2021-09-05-21-37-28.bpo-30856.jj86y0.rst b/Misc/NEWS.d/next/Library/2021-09-05-21-37-28.bpo-30856.jj86y0.rst deleted file mode 100644 index 1ac4eb672d2ec..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-05-21-37-28.bpo-30856.jj86y0.rst +++ /dev/null @@ -1,6 +0,0 @@ -:class:`unittest.TestResult` methods -:meth:`~unittest.TestResult.addFailure`, -:meth:`~unittest.TestResult.addError`, :meth:`~unittest.TestResult.addSkip` -and :meth:`~unittest.TestResult.addSubTest` are now called immediately after -raising an exception in test or finishing a subtest. Previously they were -called only after finishing the test clean up. diff --git a/Misc/NEWS.d/next/Library/2021-09-07-09-13-27.bpo-45124.Kw5AUs.rst b/Misc/NEWS.d/next/Library/2021-09-07-09-13-27.bpo-45124.Kw5AUs.rst deleted file mode 100644 index 2f6ab411b3deb..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-07-09-13-27.bpo-45124.Kw5AUs.rst +++ /dev/null @@ -1,5 +0,0 @@ -The ``bdist_msi`` command, deprecated in Python 3.9, is now removed. - -Use ``bdist_wheel`` (wheel packages) instead. - -Patch by Hugo van Kemenade. diff --git a/Misc/NEWS.d/next/Library/2021-09-07-14-27-39.bpo-45129.vXH0gw.rst b/Misc/NEWS.d/next/Library/2021-09-07-14-27-39.bpo-45129.vXH0gw.rst deleted file mode 100644 index 5ba6721923dbd..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-07-14-27-39.bpo-45129.vXH0gw.rst +++ /dev/null @@ -1,6 +0,0 @@ -Due to significant security concerns, the *reuse_address* parameter of -:meth:`asyncio.loop.create_datagram_endpoint`, disabled in Python 3.9, is -now entirely removed. This is because of the behavior of the socket option -``SO_REUSEADDR`` in UDP. - -Patch by Hugo van Kemenade. diff --git a/Misc/NEWS.d/next/Library/2021-09-07-16-33-51.bpo-45132.WI9zQY.rst b/Misc/NEWS.d/next/Library/2021-09-07-16-33-51.bpo-45132.WI9zQY.rst deleted file mode 100644 index 10f24003dc71f..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-07-16-33-51.bpo-45132.WI9zQY.rst +++ /dev/null @@ -1,5 +0,0 @@ -Remove :meth:`__getitem__` methods of -:class:`xml.dom.pulldom.DOMEventStream`, :class:`wsgiref.util.FileWrapper` -and :class:`fileinput.FileInput`, deprecated since Python 3.9. - -Patch by Hugo van Kemenade. diff --git a/Misc/NEWS.d/next/Library/2021-09-08-01-19-31.bpo-20499.tSxx8Y.rst b/Misc/NEWS.d/next/Library/2021-09-08-01-19-31.bpo-20499.tSxx8Y.rst deleted file mode 100644 index cbbe61ac4a269..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-08-01-19-31.bpo-20499.tSxx8Y.rst +++ /dev/null @@ -1 +0,0 @@ -Improve the speed and accuracy of statistics.pvariance(). diff --git a/Misc/NEWS.d/next/Library/2021-09-08-13-19-29.bpo-38371.y1kEfP.rst b/Misc/NEWS.d/next/Library/2021-09-08-13-19-29.bpo-38371.y1kEfP.rst deleted file mode 100644 index 65f35488c6f25..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-08-13-19-29.bpo-38371.y1kEfP.rst +++ /dev/null @@ -1,2 +0,0 @@ -Remove the deprecated ``split()`` method of :class:`_tkinter.TkappType`. -Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Library/2021-09-10-13-20-53.bpo-45162.2Jh-lq.rst b/Misc/NEWS.d/next/Library/2021-09-10-13-20-53.bpo-45162.2Jh-lq.rst deleted file mode 100644 index b22269d65b041..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-10-13-20-53.bpo-45162.2Jh-lq.rst +++ /dev/null @@ -1,6 +0,0 @@ -Remove many old deprecated :mod:`unittest` features: - -* "``fail*``" and "``assert*``" aliases of :class:`~unittest.TestCase` methods. -* Broken from start :class:`~unittest.TestCase` method ``assertDictContainsSubset()``. -* Ignored :meth:` TestLoader.loadTestsFromModule` parameter *use_load_tests*. -* Old alias ``_TextTestResult`` of :class:`~unittest.TextTestResult`. diff --git a/Misc/NEWS.d/next/Library/2021-09-10-21-35-53.bpo-45166.UHipXF.rst b/Misc/NEWS.d/next/Library/2021-09-10-21-35-53.bpo-45166.UHipXF.rst deleted file mode 100644 index b7242d45ea9be..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-10-21-35-53.bpo-45166.UHipXF.rst +++ /dev/null @@ -1,2 +0,0 @@ -:func:`typing.get_type_hints` now works with :data:`~typing.Final` wrapped in -:class:`~typing.ForwardRef`. diff --git a/Misc/NEWS.d/next/Library/2021-09-11-10-45-12.bpo-35474.tEY3SD.rst b/Misc/NEWS.d/next/Library/2021-09-11-10-45-12.bpo-35474.tEY3SD.rst deleted file mode 100644 index f4dd3b947a493..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-11-10-45-12.bpo-35474.tEY3SD.rst +++ /dev/null @@ -1,3 +0,0 @@ -Calling :func:`mimetypes.guess_all_extensions` with ``strict=False`` no -longer affects the result of the following call with ``strict=True``. -Also, mutating the returned list no longer affects the global state. diff --git a/Misc/NEWS.d/next/Library/2021-09-11-14-41-02.bpo-44987.Mt8DiX.rst b/Misc/NEWS.d/next/Library/2021-09-11-14-41-02.bpo-44987.Mt8DiX.rst deleted file mode 100644 index dec50d87c916c..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-11-14-41-02.bpo-44987.Mt8DiX.rst +++ /dev/null @@ -1,2 +0,0 @@ -Pure ASCII strings are now normalized in constant time by :func:`unicodedata.normalize`. -Patch by Dong-hee Na. diff --git a/Misc/NEWS.d/next/Library/2021-09-11-17-46-20.bpo-45173.UptGAn.rst b/Misc/NEWS.d/next/Library/2021-09-11-17-46-20.bpo-45173.UptGAn.rst deleted file mode 100644 index 0b29ec2364edc..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-11-17-46-20.bpo-45173.UptGAn.rst +++ /dev/null @@ -1,7 +0,0 @@ -Remove from the :mod:`configparser` module: -the :class:`SafeConfigParser` class, -the :attr:`filename` property of the :class:`ParsingError` class, -the :meth:`readfp` method of the :class:`ConfigParser` class, -deprecated since Python 3.2. - -Patch by Hugo van Kemenade. diff --git a/Misc/NEWS.d/next/Library/2021-09-11-18-44-40.bpo-21302.QxHRpR.rst b/Misc/NEWS.d/next/Library/2021-09-11-18-44-40.bpo-21302.QxHRpR.rst deleted file mode 100644 index 86f0a5ac4c2a5..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-11-18-44-40.bpo-21302.QxHRpR.rst +++ /dev/null @@ -1,2 +0,0 @@ -In Unix operating systems, :func:`time.sleep` now uses the ``clock_nanosleep()`` function, -if available, which allows to sleep for an interval specified with nanosecond precision. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-09-13-14-28-49.bpo-45168.Z1mfW4.rst b/Misc/NEWS.d/next/Library/2021-09-13-14-28-49.bpo-45168.Z1mfW4.rst deleted file mode 100644 index 4e12e7d51e2c7..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-13-14-28-49.bpo-45168.Z1mfW4.rst +++ /dev/null @@ -1 +0,0 @@ -Change :func:`dis.dis` output to omit op arg values that cannot be resolved due to ``co_consts``, ``co_names`` etc not being provided. Previously the oparg itself was repeated in the value field, which is not useful and can be confusing. diff --git a/Misc/NEWS.d/next/Library/2021-09-13-14-59-01.bpo-20524.PMQ1Fh.rst b/Misc/NEWS.d/next/Library/2021-09-13-14-59-01.bpo-20524.PMQ1Fh.rst deleted file mode 100644 index 9a41c8e64f7e3..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-13-14-59-01.bpo-20524.PMQ1Fh.rst +++ /dev/null @@ -1,3 +0,0 @@ -Improves error messages on ``.format()`` operation for ``str``, ``float``, -``int``, and ``complex``. New format now shows the problematic pattern and -the object type. diff --git a/Misc/NEWS.d/next/Library/2021-09-13-19-32-58.bpo-42135.1ZAHqR.rst b/Misc/NEWS.d/next/Library/2021-09-13-19-32-58.bpo-42135.1ZAHqR.rst deleted file mode 100644 index 983b46e140e74..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-13-19-32-58.bpo-42135.1ZAHqR.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix typo: ``importlib.find_loader`` is really slated for removal in Python 3.12 not 3.10, like the others in GH-25169. - -Patch by Hugo van Kemenade. diff --git a/Misc/NEWS.d/next/Library/2021-09-16-19-02-14.bpo-45225.xmKV4i.rst b/Misc/NEWS.d/next/Library/2021-09-16-19-02-14.bpo-45225.xmKV4i.rst deleted file mode 100644 index 734fdd9b007d6..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-16-19-02-14.bpo-45225.xmKV4i.rst +++ /dev/null @@ -1 +0,0 @@ -use map function instead of genexpr in capwords. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-09-17-09-59-33.bpo-45228.WV1dcT.rst b/Misc/NEWS.d/next/Library/2021-09-17-09-59-33.bpo-45228.WV1dcT.rst deleted file mode 100644 index 9336c0aed92bc..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-17-09-59-33.bpo-45228.WV1dcT.rst +++ /dev/null @@ -1 +0,0 @@ -Fix stack buffer overflow in parsing J1939 network address. diff --git a/Misc/NEWS.d/next/Library/2021-09-17-11-20-55.bpo-45234.qUcTVt.rst b/Misc/NEWS.d/next/Library/2021-09-17-11-20-55.bpo-45234.qUcTVt.rst deleted file mode 100644 index 3817b5de6449d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-17-11-20-55.bpo-45234.qUcTVt.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fixed a regression in :func:`~shutil.copyfile`, :func:`~shutil.copy`, -:func:`~shutil.copy2` raising :exc:`FileNotFoundError` when source is a -directory, which should raise :exc:`IsADirectoryError` diff --git a/Misc/NEWS.d/next/Library/2021-09-17-15-58-53.bpo-45183.Vv_vch.rst b/Misc/NEWS.d/next/Library/2021-09-17-15-58-53.bpo-45183.Vv_vch.rst deleted file mode 100644 index f3194b34318f4..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-17-15-58-53.bpo-45183.Vv_vch.rst +++ /dev/null @@ -1,3 +0,0 @@ -Have zipimport.zipimporter.find_spec() not raise an exception when the underlying zip -file has been deleted and the internal cache has been reset via -invalidate_cache(). diff --git a/Misc/NEWS.d/next/Library/2021-09-17-16-55-37.bpo-45235.sXnmPA.rst b/Misc/NEWS.d/next/Library/2021-09-17-16-55-37.bpo-45235.sXnmPA.rst deleted file mode 100644 index 871ec5281d334..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-17-16-55-37.bpo-45235.sXnmPA.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix an issue where argparse would not preserve values in a provided namespace -when using a subparser with defaults. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-09-18-13-14-57.bpo-36674.a2k5Zb.rst b/Misc/NEWS.d/next/Library/2021-09-18-13-14-57.bpo-36674.a2k5Zb.rst deleted file mode 100644 index bc8c9247b080d..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-18-13-14-57.bpo-36674.a2k5Zb.rst +++ /dev/null @@ -1,2 +0,0 @@ -:meth:`unittest.TestCase.debug` raises now a :class:`unittest.SkipTest` if -the class or the test method are decorated with the skipping decorator. diff --git a/Misc/NEWS.d/next/Library/2021-09-18-16-56-33.bpo-45238.Hng_9V.rst b/Misc/NEWS.d/next/Library/2021-09-18-16-56-33.bpo-45238.Hng_9V.rst deleted file mode 100644 index 857f315c520bb..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-18-16-56-33.bpo-45238.Hng_9V.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix :meth:`unittest.IsolatedAsyncioTestCase.debug`: it runs now asynchronous -methods and callbacks. diff --git a/Misc/NEWS.d/next/Library/2021-09-20-22-46-40.bpo-21302.h56430.rst b/Misc/NEWS.d/next/Library/2021-09-20-22-46-40.bpo-21302.h56430.rst deleted file mode 100644 index 07f18d4a9074a..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-20-22-46-40.bpo-21302.h56430.rst +++ /dev/null @@ -1,4 +0,0 @@ -On Windows, :func:`time.sleep` now uses a waitable timer which has a resolution -of 100 nanoseconds (10\ :sup:`-7` seconds). Previously, it had a resolution of -1 millisecond (10\ :sup:`-3` seconds). -Patch by Livius and Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-09-22-23-56-15.bpo-21302.vvQ3Su.rst b/Misc/NEWS.d/next/Library/2021-09-22-23-56-15.bpo-21302.vvQ3Su.rst deleted file mode 100644 index 52ee8d7cc64f1..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-22-23-56-15.bpo-21302.vvQ3Su.rst +++ /dev/null @@ -1 +0,0 @@ -In Unix operating systems, :func:`time.sleep` now uses the ``nanosleep()`` function, if ``clock_nanosleep()`` is not available but ``nanosleep()`` is available. ``nanosleep()`` allows to sleep with nanosecond precision. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Library/2021-09-23-22-17-26.bpo-45274.gPpa4E.rst b/Misc/NEWS.d/next/Library/2021-09-23-22-17-26.bpo-45274.gPpa4E.rst deleted file mode 100644 index 94d06cef89b7b..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-23-22-17-26.bpo-45274.gPpa4E.rst +++ /dev/null @@ -1,5 +0,0 @@ -Fix a race condition in the :meth:`Thread.join() ` -method of the :mod:`threading` module. If the function is interrupted by a -signal and the signal handler raises an exception, make sure that the thread -remains in a consistent state to prevent a deadlock. Patch by Victor -Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-09-24-17-20-23.bpo-1596321.3nhPUk.rst b/Misc/NEWS.d/next/Library/2021-09-24-17-20-23.bpo-1596321.3nhPUk.rst deleted file mode 100644 index 61a3e5abf395e..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-24-17-20-23.bpo-1596321.3nhPUk.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix the :func:`threading._shutdown` function when the :mod:`threading` module -was imported first from a thread different than the main thread: no longer log -an error at Python exit. diff --git a/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst b/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst deleted file mode 100644 index d8a4f9507c189..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-09-30-23-00-18.bpo-41710.svuloZ.rst +++ /dev/null @@ -1,5 +0,0 @@ -On Unix, if the ``sem_clockwait()`` function is available in the C library -(glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses the -monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather than -using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected by -system clock changes. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst b/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst deleted file mode 100644 index b4bedbc278edf..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-10-01-13-09-53.bpo-45329.9iMYaO.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix freed memory access in :class:`pyexpat.xmlparser` when building it with an -installed expat library <= 2.2.0. diff --git a/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst b/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst deleted file mode 100644 index 045489be81a19..0000000000000 --- a/Misc/NEWS.d/next/Library/2021-10-05-11-03-48.bpo-45371.NOwcDJ.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix clang rpath issue in :mod:`distutils`. The UnixCCompiler now uses -correct clang option to add a runtime library directory (rpath) to a shared -library. diff --git a/Misc/NEWS.d/next/Security/2021-05-05-17-37-04.bpo-44022.bS3XJ9.rst b/Misc/NEWS.d/next/Security/2021-05-05-17-37-04.bpo-44022.bS3XJ9.rst deleted file mode 100644 index 9669fc5ef37dd..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-05-05-17-37-04.bpo-44022.bS3XJ9.rst +++ /dev/null @@ -1,2 +0,0 @@ -:mod:`http.client` now avoids infinitely reading potential HTTP headers after a -``100 Continue`` status response from the server. diff --git a/Misc/NEWS.d/next/Security/2021-05-08-11-50-46.bpo-43124.2CTM6M.rst b/Misc/NEWS.d/next/Security/2021-05-08-11-50-46.bpo-43124.2CTM6M.rst deleted file mode 100644 index e897d6cd3641d..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-05-08-11-50-46.bpo-43124.2CTM6M.rst +++ /dev/null @@ -1,2 +0,0 @@ -Made the internal ``putcmd`` function in :mod:`smtplib` sanitize input for -presence of ``\r`` and ``\n`` characters to avoid (unlikely) command injection. diff --git a/Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst b/Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst deleted file mode 100644 index e32563d2535c7..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-06-29-02-45-53.bpo-44394.A220N1.rst +++ /dev/null @@ -1,3 +0,0 @@ -Update the vendored copy of libexpat to 2.4.1 (from 2.2.8) to get the fix -for the CVE-2013-0340 "Billion Laughs" vulnerability. This copy is most used -on Windows and macOS. diff --git a/Misc/NEWS.d/next/Security/2021-06-29-23-40-22.bpo-41180.uTWHv_.rst b/Misc/NEWS.d/next/Security/2021-06-29-23-40-22.bpo-41180.uTWHv_.rst deleted file mode 100644 index 88b70c7cea261..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-06-29-23-40-22.bpo-41180.uTWHv_.rst +++ /dev/null @@ -1,5 +0,0 @@ -Add auditing events to the :mod:`marshal` module, and stop raising -``code.__init__`` events for every unmarshalled code object. Directly -instantiated code objects will continue to raise an event, and audit event -handlers should inspect or collect the raw marshal data. This reduces a -significant performance overhead when loading from ``.pyc`` files. diff --git a/Misc/NEWS.d/next/Security/2021-07-25-20-04-54.bpo-44600.0WMldg.rst b/Misc/NEWS.d/next/Security/2021-07-25-20-04-54.bpo-44600.0WMldg.rst deleted file mode 100644 index ea4e04f6da911..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-07-25-20-04-54.bpo-44600.0WMldg.rst +++ /dev/null @@ -1 +0,0 @@ -Fix incorrect line numbers while tracing some failed patterns in :ref:`match ` statements. Patch by Charles Burkland. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Security/2021-08-29-12-39-44.bpo-42278.jvmQz_.rst b/Misc/NEWS.d/next/Security/2021-08-29-12-39-44.bpo-42278.jvmQz_.rst deleted file mode 100644 index db880cd9026da..0000000000000 --- a/Misc/NEWS.d/next/Security/2021-08-29-12-39-44.bpo-42278.jvmQz_.rst +++ /dev/null @@ -1,2 +0,0 @@ -Replaced usage of :func:`tempfile.mktemp` with -:class:`~tempfile.TemporaryDirectory` to avoid a potential race condition. diff --git a/Misc/NEWS.d/next/Tests/2019-09-25-18-10-10.bpo-30256.A5i76Q.rst b/Misc/NEWS.d/next/Tests/2019-09-25-18-10-10.bpo-30256.A5i76Q.rst deleted file mode 100644 index 4a7cfd52fcea2..0000000000000 --- a/Misc/NEWS.d/next/Tests/2019-09-25-18-10-10.bpo-30256.A5i76Q.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add test for nested queues when using ``multiprocessing`` shared objects -``AutoProxy[Queue]`` inside ``ListProxy`` and ``DictProxy`` diff --git a/Misc/NEWS.d/next/Tests/2020-10-25-19-20-26.bpo-35753.2LT-hO.rst b/Misc/NEWS.d/next/Tests/2020-10-25-19-20-26.bpo-35753.2LT-hO.rst deleted file mode 100644 index eddfc25906da9..0000000000000 --- a/Misc/NEWS.d/next/Tests/2020-10-25-19-20-26.bpo-35753.2LT-hO.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix crash in doctest when doctest parses modules that include unwrappable -functions by skipping those functions. diff --git a/Misc/NEWS.d/next/Tests/2021-05-04-18-10-57.bpo-42083.EMS2TK.rst b/Misc/NEWS.d/next/Tests/2021-05-04-18-10-57.bpo-42083.EMS2TK.rst deleted file mode 100644 index 8457508237a88..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-05-04-18-10-57.bpo-42083.EMS2TK.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add test to check that ``PyStructSequence_NewType`` accepts a -``PyStructSequence_Desc`` with ``doc`` field set to ``NULL``. diff --git a/Misc/NEWS.d/next/Tests/2021-05-07-15-46-04.bpo-31904.8dk3la.rst b/Misc/NEWS.d/next/Tests/2021-05-07-15-46-04.bpo-31904.8dk3la.rst deleted file mode 100644 index 334878f91ddac..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-05-07-15-46-04.bpo-31904.8dk3la.rst +++ /dev/null @@ -1 +0,0 @@ -Ignore error string case in test_file_not_exists(). diff --git a/Misc/NEWS.d/next/Tests/2021-05-14-14-13-44.bpo-44131.YPizoC.rst b/Misc/NEWS.d/next/Tests/2021-05-14-14-13-44.bpo-44131.YPizoC.rst deleted file mode 100644 index a646acf8e4443..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-05-14-14-13-44.bpo-44131.YPizoC.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add test_frozenmain to test_embed to test the :c:func:`Py_FrozenMain` C -function. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-06-02-17-41-42.bpo-43921.xP7yZ4.rst b/Misc/NEWS.d/next/Tests/2021-06-02-17-41-42.bpo-43921.xP7yZ4.rst deleted file mode 100644 index 83146c7852467..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-02-17-41-42.bpo-43921.xP7yZ4.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix test_pha_required_nocert() of test_ssl: catch two more EOF cases (when -the ``recv()`` method returns an empty string). Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-06-03-03-29-34.bpo-43921.nwH1FS.rst b/Misc/NEWS.d/next/Tests/2021-06-03-03-29-34.bpo-43921.nwH1FS.rst deleted file mode 100644 index 30e0fadd66125..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-03-03-29-34.bpo-43921.nwH1FS.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix test_ssl.test_wrong_cert_tls13(): use ``suppress_ragged_eofs=False``, -since ``read()`` can raise :exc:`ssl.SSLEOFError` on Windows. Patch by -Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-06-09-15-32-05.bpo-44364.zu9Zee.rst b/Misc/NEWS.d/next/Tests/2021-06-09-15-32-05.bpo-44364.zu9Zee.rst deleted file mode 100644 index 12b80e8e6533b..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-09-15-32-05.bpo-44364.zu9Zee.rst +++ /dev/null @@ -1 +0,0 @@ -Add non integral tests for :func:`math.sqrt` function. diff --git a/Misc/NEWS.d/next/Tests/2021-06-10-11-19-43.bpo-44363.-K9jD0.rst b/Misc/NEWS.d/next/Tests/2021-06-10-11-19-43.bpo-44363.-K9jD0.rst deleted file mode 100644 index 28468cbd2b682..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-10-11-19-43.bpo-44363.-K9jD0.rst +++ /dev/null @@ -1,2 +0,0 @@ -Account for address sanitizer in test_capi. test_capi now passes when run -GCC address sanitizer. diff --git a/Misc/NEWS.d/next/Tests/2021-06-18-15-19-35.bpo-44451.aj5pqE.rst b/Misc/NEWS.d/next/Tests/2021-06-18-15-19-35.bpo-44451.aj5pqE.rst deleted file mode 100644 index 0f635cfe18d14..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-18-15-19-35.bpo-44451.aj5pqE.rst +++ /dev/null @@ -1,3 +0,0 @@ -Reset ``DeprecationWarning`` filters in -``test.test_importlib.test_metadata_api.APITests.test_entry_points_by_index`` -to avoid ``StopIteration`` error if ``DeprecationWarnings`` are ignored. diff --git a/Misc/NEWS.d/next/Tests/2021-06-21-17-53-41.bpo-44287.YON57s.rst b/Misc/NEWS.d/next/Tests/2021-06-21-17-53-41.bpo-44287.YON57s.rst deleted file mode 100644 index 66b3afe139aa8..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-21-17-53-41.bpo-44287.YON57s.rst +++ /dev/null @@ -1,4 +0,0 @@ -Fix asyncio test_popen() of test_windows_utils by using a longer timeout. -Use military grade battle-tested :data:`test.support.SHORT_TIMEOUT` timeout -rather than a hardcoded timeout of 10 seconds: it's 30 seconds by default, but -it is made longer on slow buildbots. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-06-26-18-37-36.bpo-44515.e9fO6f.rst b/Misc/NEWS.d/next/Tests/2021-06-26-18-37-36.bpo-44515.e9fO6f.rst deleted file mode 100644 index d2867b6e89f87..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-06-26-18-37-36.bpo-44515.e9fO6f.rst +++ /dev/null @@ -1,2 +0,0 @@ -Adjust recently added contextlib tests to avoid assuming the use of a -refcounted GC diff --git a/Misc/NEWS.d/next/Tests/2021-07-16-14-02-33.bpo-44647.5LzqIy.rst b/Misc/NEWS.d/next/Tests/2021-07-16-14-02-33.bpo-44647.5LzqIy.rst deleted file mode 100644 index e4b2be2b40999..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-07-16-14-02-33.bpo-44647.5LzqIy.rst +++ /dev/null @@ -1,4 +0,0 @@ -Added a permanent Unicode-valued environment variable to regression tests to -ensure they handle this use case in the future. If your test environment -breaks because of that, report a bug to us, and temporarily set -PYTHONREGRTEST_UNICODE_GUARD=0 in your test environment. diff --git a/Misc/NEWS.d/next/Tests/2021-07-17-11-41-20.bpo-42095.kCB7oj.rst b/Misc/NEWS.d/next/Tests/2021-07-17-11-41-20.bpo-42095.kCB7oj.rst deleted file mode 100644 index bf7bc5b2ff449..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-07-17-11-41-20.bpo-42095.kCB7oj.rst +++ /dev/null @@ -1,2 +0,0 @@ -Added interop tests for Apple plists: generate plist files with Python -plistlib and parse with Apple plutil; and the other way round. diff --git a/Misc/NEWS.d/next/Tests/2021-07-22-16-38-39.bpo-44708.SYNaac.rst b/Misc/NEWS.d/next/Tests/2021-07-22-16-38-39.bpo-44708.SYNaac.rst deleted file mode 100644 index 8b26c4de64b98..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-07-22-16-38-39.bpo-44708.SYNaac.rst +++ /dev/null @@ -1,2 +0,0 @@ -Regression tests, when run with -w, are now re-running only the affected -test methods instead of re-running the entire test file. diff --git a/Misc/NEWS.d/next/Tests/2021-07-24-20-09-15.bpo-44734.KKsNOV.rst b/Misc/NEWS.d/next/Tests/2021-07-24-20-09-15.bpo-44734.KKsNOV.rst deleted file mode 100644 index 94e9ce08f4bf6..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-07-24-20-09-15.bpo-44734.KKsNOV.rst +++ /dev/null @@ -1 +0,0 @@ -Fixed floating point precision issue in turtle tests. diff --git a/Misc/NEWS.d/next/Tests/2021-08-06-00-07-15.bpo-40928.aIwb6G.rst b/Misc/NEWS.d/next/Tests/2021-08-06-00-07-15.bpo-40928.aIwb6G.rst deleted file mode 100644 index c9a5e1b01e58a..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-06-00-07-15.bpo-40928.aIwb6G.rst +++ /dev/null @@ -1,2 +0,0 @@ -Notify users running test_decimal regression tests on macOS of potential -harmless "malloc can't allocate region" messages spewed by test_decimal. diff --git a/Misc/NEWS.d/next/Tests/2021-08-06-18-36-04.bpo-44852.sUL8YX.rst b/Misc/NEWS.d/next/Tests/2021-08-06-18-36-04.bpo-44852.sUL8YX.rst deleted file mode 100644 index 41b5c2fb51c5c..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-06-18-36-04.bpo-44852.sUL8YX.rst +++ /dev/null @@ -1,2 +0,0 @@ -Add ability to wholesale silence DeprecationWarnings while running the -regression test suite. diff --git a/Misc/NEWS.d/next/Tests/2021-08-13-12-11-06.bpo-44891.T9_mBT.rst b/Misc/NEWS.d/next/Tests/2021-08-13-12-11-06.bpo-44891.T9_mBT.rst deleted file mode 100644 index 2f83389ec1585..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-13-12-11-06.bpo-44891.T9_mBT.rst +++ /dev/null @@ -1,2 +0,0 @@ -Tests were added to clarify :func:`id` is preserved when ``obj * 1`` is used -on :class:`str` and :class:`bytes` objects. Patch by Nikita Sobolev. diff --git a/Misc/NEWS.d/next/Tests/2021-08-18-18-30-12.bpo-44949.VE5ENv.rst b/Misc/NEWS.d/next/Tests/2021-08-18-18-30-12.bpo-44949.VE5ENv.rst deleted file mode 100644 index 7fdf1810b165e..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-18-18-30-12.bpo-44949.VE5ENv.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix auto history tests of test_readline: sometimes, the newline character is -not written at the end, so don't expect it in the output. diff --git a/Misc/NEWS.d/next/Tests/2021-08-26-14-20-44.bpo-45011.mQZdXU.rst b/Misc/NEWS.d/next/Tests/2021-08-26-14-20-44.bpo-45011.mQZdXU.rst deleted file mode 100644 index 64e701e2f29f4..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-26-14-20-44.bpo-45011.mQZdXU.rst +++ /dev/null @@ -1,3 +0,0 @@ -Made tests relying on the :mod:`_asyncio` C extension module optional to -allow running on alternative Python implementations. Patch by Serhiy -Storchaka. diff --git a/Misc/NEWS.d/next/Tests/2021-08-27-22-37-19.bpo-25130.ig4oJe.rst b/Misc/NEWS.d/next/Tests/2021-08-27-22-37-19.bpo-25130.ig4oJe.rst deleted file mode 100644 index 43ce68bef4609..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-27-22-37-19.bpo-25130.ig4oJe.rst +++ /dev/null @@ -1 +0,0 @@ -Add calls of :func:`gc.collect` in tests to support PyPy. diff --git a/Misc/NEWS.d/next/Tests/2021-08-30-11-54-14.bpo-45042.QMz3X8.rst b/Misc/NEWS.d/next/Tests/2021-08-30-11-54-14.bpo-45042.QMz3X8.rst deleted file mode 100644 index e2c0dffced96e..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-08-30-11-54-14.bpo-45042.QMz3X8.rst +++ /dev/null @@ -1 +0,0 @@ -Fixes that test classes decorated with ``@hashlib_helper.requires_hashdigest`` were skipped all the time. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tests/2021-09-01-17-17-44.bpo-44895.kV7H77.rst b/Misc/NEWS.d/next/Tests/2021-09-01-17-17-44.bpo-44895.kV7H77.rst deleted file mode 100644 index 038466f8d6a4f..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-01-17-17-44.bpo-44895.kV7H77.rst +++ /dev/null @@ -1,5 +0,0 @@ -libregrtest now clears the type cache later to reduce the risk of false alarm -when checking for reference leaks. Previously, the type cache was cleared too -early and libregrtest raised a false alarm about reference leaks under very -specific conditions. -Patch by Irit Katriel and Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-09-06-19-00-29.bpo-45052.yrOK3J.rst b/Misc/NEWS.d/next/Tests/2021-09-06-19-00-29.bpo-45052.yrOK3J.rst deleted file mode 100644 index 5c2e4f3e0e67b..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-06-19-00-29.bpo-45052.yrOK3J.rst +++ /dev/null @@ -1,7 +0,0 @@ -``WithProcessesTestSharedMemory.test_shared_memory_basics`` test was -ignored, because ``self.assertEqual(sms.size, sms2.size)`` line was failing. -It is now removed and test is unskipped. - -The main motivation for this line to be removed from the test is that the -``size`` of ``SharedMemory`` is not ever guaranteed to be the same. It is -decided by the platform. diff --git a/Misc/NEWS.d/next/Tests/2021-09-08-13-01-37.bpo-44860.qXd0kx.rst b/Misc/NEWS.d/next/Tests/2021-09-08-13-01-37.bpo-44860.qXd0kx.rst deleted file mode 100644 index 153a9c55733fb..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-08-13-01-37.bpo-44860.qXd0kx.rst +++ /dev/null @@ -1,2 +0,0 @@ -Update ``test_sysconfig.test_user_similar()`` for the posix_user scheme: -``platlib`` doesn't use :data:`sys.platlibdir`. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst b/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst deleted file mode 100644 index 5dfbe0e5db463..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-11-22-08-18.bpo-45125.FVSzs2.rst +++ /dev/null @@ -1,2 +0,0 @@ -Improves pickling tests and docs of ``SharedMemory`` and ``SharableList`` -objects. diff --git a/Misc/NEWS.d/next/Tests/2021-09-13-00-28-17.bpo-45156.8oomV3.rst b/Misc/NEWS.d/next/Tests/2021-09-13-00-28-17.bpo-45156.8oomV3.rst deleted file mode 100644 index b2094b5765331..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-13-00-28-17.bpo-45156.8oomV3.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixes infinite loop on :func:`unittest.mock.seal` of mocks created by -:func:`~unittest.create_autospec`. diff --git a/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst b/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst deleted file mode 100644 index 16a1f4440483c..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst +++ /dev/null @@ -1,3 +0,0 @@ -Fix test_readline.test_nonascii(): sometimes, the newline character is not -written at the end, so don't expect it in the output. Patch by Victor -Stinner. diff --git a/Misc/NEWS.d/next/Tests/2021-09-14-14-54-04.bpo-45185.qFx5I6.rst b/Misc/NEWS.d/next/Tests/2021-09-14-14-54-04.bpo-45185.qFx5I6.rst deleted file mode 100644 index e723f24759e9e..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-14-14-54-04.bpo-45185.qFx5I6.rst +++ /dev/null @@ -1 +0,0 @@ -Enables ``TestEnumerations`` test cases in ``test_ssl`` suite. diff --git a/Misc/NEWS.d/next/Tests/2021-09-15-23-32-39.bpo-45209.55ntL5.rst b/Misc/NEWS.d/next/Tests/2021-09-15-23-32-39.bpo-45209.55ntL5.rst deleted file mode 100644 index 4c3bed0983b89..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-15-23-32-39.bpo-45209.55ntL5.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``UserWarning: resource_tracker`` warning in -``_test_multiprocessing._TestSharedMemory.test_shared_memory_cleaned_after_process_termination`` diff --git a/Misc/NEWS.d/next/Tests/2021-09-16-17-22-35.bpo-45128.Jz6fl2.rst b/Misc/NEWS.d/next/Tests/2021-09-16-17-22-35.bpo-45128.Jz6fl2.rst deleted file mode 100644 index b50eb32b3faa8..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-16-17-22-35.bpo-45128.Jz6fl2.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix ``test_multiprocessing_fork`` failure due to ``test_logging`` and -``sys.modules`` manipulation. diff --git a/Misc/NEWS.d/next/Tests/2021-09-24-10-41-49.bpo-45269.8jKEr8.rst b/Misc/NEWS.d/next/Tests/2021-09-24-10-41-49.bpo-45269.8jKEr8.rst deleted file mode 100644 index 72dd9471134ff..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-24-10-41-49.bpo-45269.8jKEr8.rst +++ /dev/null @@ -1 +0,0 @@ -Cover case when invalid ``markers`` type is supplied to ``c_make_encoder``. diff --git a/Misc/NEWS.d/next/Tests/2021-09-25-11-05-31.bpo-45280.3MA6lC.rst b/Misc/NEWS.d/next/Tests/2021-09-25-11-05-31.bpo-45280.3MA6lC.rst deleted file mode 100644 index 71691f5ba2b89..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-25-11-05-31.bpo-45280.3MA6lC.rst +++ /dev/null @@ -1 +0,0 @@ -Add a test case for empty :class:`typing.NamedTuple`. diff --git a/Misc/NEWS.d/next/Tests/2021-09-30-16-54-39.bpo-40173.J_slCw.rst b/Misc/NEWS.d/next/Tests/2021-09-30-16-54-39.bpo-40173.J_slCw.rst deleted file mode 100644 index 21671473c16cc..0000000000000 --- a/Misc/NEWS.d/next/Tests/2021-09-30-16-54-39.bpo-40173.J_slCw.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fix :func:`test.support.import_helper.import_fresh_module`. - diff --git a/Misc/NEWS.d/next/Tools-Demos/2020-02-25-18-22-09.bpo-20291.AyrDiZ.rst b/Misc/NEWS.d/next/Tools-Demos/2020-02-25-18-22-09.bpo-20291.AyrDiZ.rst deleted file mode 100644 index c64c5488bfaa1..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2020-02-25-18-22-09.bpo-20291.AyrDiZ.rst +++ /dev/null @@ -1 +0,0 @@ -Added support for variadic positional parameters in Argument Clinic. diff --git a/Misc/NEWS.d/next/Tools-Demos/2021-05-08-13-57-00.bpo-44074.F09kCK.rst b/Misc/NEWS.d/next/Tools-Demos/2021-05-08-13-57-00.bpo-44074.F09kCK.rst deleted file mode 100644 index 8bccb080a5418..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2021-05-08-13-57-00.bpo-44074.F09kCK.rst +++ /dev/null @@ -1 +0,0 @@ -Make patchcheck automatically detect the correct base branch name (previously it was hardcoded to 'master') \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tools-Demos/2021-07-01-22-21-25.bpo-43425.t65len.rst b/Misc/NEWS.d/next/Tools-Demos/2021-07-01-22-21-25.bpo-43425.t65len.rst deleted file mode 100644 index b9ce6c467f90b..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2021-07-01-22-21-25.bpo-43425.t65len.rst +++ /dev/null @@ -1,3 +0,0 @@ -Removed the 'test2to3' demo project that demonstrated using lib2to3 -to support Python 2.x and Python 3.x from a single source in -a distutils package. Patch by Dong-hee Na diff --git a/Misc/NEWS.d/next/Tools-Demos/2021-08-22-11-45-31.bpo-44978.QupKV3.rst b/Misc/NEWS.d/next/Tools-Demos/2021-08-22-11-45-31.bpo-44978.QupKV3.rst deleted file mode 100644 index c7a844c1230e2..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2021-08-22-11-45-31.bpo-44978.QupKV3.rst +++ /dev/null @@ -1 +0,0 @@ -Allow the Argument Clinic tool to handle ``__complex__`` special methods. diff --git a/Misc/NEWS.d/next/Tools-Demos/2021-08-26-11-57-31.bpo-44967.UT1RMV.rst b/Misc/NEWS.d/next/Tools-Demos/2021-08-26-11-57-31.bpo-44967.UT1RMV.rst deleted file mode 100644 index 564a5c7d76894..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2021-08-26-11-57-31.bpo-44967.UT1RMV.rst +++ /dev/null @@ -1 +0,0 @@ -pydoc now returns a non-zero status code when a module cannot be found. \ No newline at end of file diff --git a/Misc/NEWS.d/next/Tools-Demos/2021-09-14-11-44-26.bpo-44786.DU0LC0.rst b/Misc/NEWS.d/next/Tools-Demos/2021-09-14-11-44-26.bpo-44786.DU0LC0.rst deleted file mode 100644 index 96ebf2c33cc5a..0000000000000 --- a/Misc/NEWS.d/next/Tools-Demos/2021-09-14-11-44-26.bpo-44786.DU0LC0.rst +++ /dev/null @@ -1 +0,0 @@ -Fix a warning in regular expression in the c-analyzer script. diff --git a/Misc/NEWS.d/next/Windows/2020-04-13-15-20-28.bpo-40263.1KKEbu.rst b/Misc/NEWS.d/next/Windows/2020-04-13-15-20-28.bpo-40263.1KKEbu.rst deleted file mode 100644 index 0c31606d4928c..0000000000000 --- a/Misc/NEWS.d/next/Windows/2020-04-13-15-20-28.bpo-40263.1KKEbu.rst +++ /dev/null @@ -1,3 +0,0 @@ -This is a follow-on bug from https://bugs.python.org/issue26903. Once that -is applied we run into an off-by-one assertion problem. The assert was not -correct. diff --git a/Misc/NEWS.d/next/Windows/2021-01-01-21-21-03.bpo-42686.G_f-TC.rst b/Misc/NEWS.d/next/Windows/2021-01-01-21-21-03.bpo-42686.G_f-TC.rst deleted file mode 100644 index 2fcf6e946ec83..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-01-01-21-21-03.bpo-42686.G_f-TC.rst +++ /dev/null @@ -1 +0,0 @@ -Build :mod:`sqlite3` with math functions enabled. Patch by Erlend E. Aasland. diff --git a/Misc/NEWS.d/next/Windows/2021-06-06-16-36-13.bpo-41299.Rg-vb_.rst b/Misc/NEWS.d/next/Windows/2021-06-06-16-36-13.bpo-41299.Rg-vb_.rst deleted file mode 100644 index 1104882b1ba30..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-06-06-16-36-13.bpo-41299.Rg-vb_.rst +++ /dev/null @@ -1 +0,0 @@ -Fix 16 milliseconds jitter when using timeouts in :mod:`threading`, such as with :meth:`threading.Lock.acquire` or :meth:`threading.Condition.wait`. diff --git a/Misc/NEWS.d/next/Windows/2021-07-07-21-07-18.bpo-44582.4Mm6Hh.rst b/Misc/NEWS.d/next/Windows/2021-07-07-21-07-18.bpo-44582.4Mm6Hh.rst deleted file mode 100644 index f79c88931c531..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-07-07-21-07-18.bpo-44582.4Mm6Hh.rst +++ /dev/null @@ -1,2 +0,0 @@ -Accelerate speed of :mod:`mimetypes` initialization using a native -implementation of the registry scan. diff --git a/Misc/NEWS.d/next/Windows/2021-07-13-15-32-49.bpo-44572.gXvhDc.rst b/Misc/NEWS.d/next/Windows/2021-07-13-15-32-49.bpo-44572.gXvhDc.rst deleted file mode 100644 index 6e074c59b8445..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-07-13-15-32-49.bpo-44572.gXvhDc.rst +++ /dev/null @@ -1 +0,0 @@ -Avoid consuming standard input in the :mod:`platform` module \ No newline at end of file diff --git a/Misc/NEWS.d/next/Windows/2021-08-06-10-11-07.bpo-44848.ib3Jcz.rst b/Misc/NEWS.d/next/Windows/2021-08-06-10-11-07.bpo-44848.ib3Jcz.rst deleted file mode 100644 index 6eadfedda865e..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-08-06-10-11-07.bpo-44848.ib3Jcz.rst +++ /dev/null @@ -1 +0,0 @@ -Upgrade Windows installer to use SQLite 3.36.0. diff --git a/Misc/NEWS.d/next/Windows/2021-08-27-23-50-02.bpo-45007.NIBlVG.rst b/Misc/NEWS.d/next/Windows/2021-08-27-23-50-02.bpo-45007.NIBlVG.rst deleted file mode 100644 index fa076ee4c8b8a..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-08-27-23-50-02.bpo-45007.NIBlVG.rst +++ /dev/null @@ -1 +0,0 @@ -Update to OpenSSL 1.1.1l in Windows build diff --git a/Misc/NEWS.d/next/Windows/2021-09-03-18-05-21.bpo-45022.bgpD_r.rst b/Misc/NEWS.d/next/Windows/2021-09-03-18-05-21.bpo-45022.bgpD_r.rst deleted file mode 100644 index 8c19faad77176..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-09-03-18-05-21.bpo-45022.bgpD_r.rst +++ /dev/null @@ -1 +0,0 @@ -Update Windows release to include libffi 3.4.2 diff --git a/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst b/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst deleted file mode 100644 index c72164373abe6..0000000000000 --- a/Misc/NEWS.d/next/Windows/2021-10-05-12-41-53.bpo-45375.CohPP-.rst +++ /dev/null @@ -1,2 +0,0 @@ -Fixes an assertion failure due to searching for the standard library in -unnormalised paths. diff --git a/Misc/NEWS.d/next/macOS/2021-03-29-21-11-23.bpo-34932.f3Hdyd.rst b/Misc/NEWS.d/next/macOS/2021-03-29-21-11-23.bpo-34932.f3Hdyd.rst deleted file mode 100644 index d3a52c9fc37b5..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-03-29-21-11-23.bpo-34932.f3Hdyd.rst +++ /dev/null @@ -1 +0,0 @@ -Add socket.TCP_KEEPALIVE support for macOS. Patch by Shane Harvey. diff --git a/Misc/NEWS.d/next/macOS/2021-05-24-21-15-41.bpo-43109.npKJ9c.rst b/Misc/NEWS.d/next/macOS/2021-05-24-21-15-41.bpo-43109.npKJ9c.rst deleted file mode 100644 index bb4d24fa95513..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-05-24-21-15-41.bpo-43109.npKJ9c.rst +++ /dev/null @@ -1,2 +0,0 @@ -Allow --with-lto configure option to work with Apple-supplied Xcode or -Command Line Tools. diff --git a/Misc/NEWS.d/next/macOS/2021-07-12-15-42-02.bpo-41972.yUjE8j.rst b/Misc/NEWS.d/next/macOS/2021-07-12-15-42-02.bpo-41972.yUjE8j.rst deleted file mode 100644 index 6c70c07669f9c..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-07-12-15-42-02.bpo-41972.yUjE8j.rst +++ /dev/null @@ -1,2 +0,0 @@ -The framework build's user header path in sysconfig is changed to add a -'pythonX.Y' component to match distutils's behavior. diff --git a/Misc/NEWS.d/next/macOS/2021-07-20-22-27-01.bpo-44689.mmT_xH.rst b/Misc/NEWS.d/next/macOS/2021-07-20-22-27-01.bpo-44689.mmT_xH.rst deleted file mode 100644 index b1e878d1ee44a..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-07-20-22-27-01.bpo-44689.mmT_xH.rst +++ /dev/null @@ -1,5 +0,0 @@ - :meth:`ctypes.util.find_library` now works correctly on macOS 11 Big Sur - even if Python is built on an older version of macOS. Previously, when - built on older macOS systems, ``find_library`` was not able to find - macOS system libraries when running on Big Sur due to changes in - how system libraries are stored. diff --git a/Misc/NEWS.d/next/macOS/2021-08-06-10-08-41.bpo-44848.0uYXsE.rst b/Misc/NEWS.d/next/macOS/2021-08-06-10-08-41.bpo-44848.0uYXsE.rst deleted file mode 100644 index 7e9c41a8e9a48..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-08-06-10-08-41.bpo-44848.0uYXsE.rst +++ /dev/null @@ -1 +0,0 @@ -Update macOS installer to use SQLite 3.36.0. diff --git a/Misc/NEWS.d/next/macOS/2021-08-27-16-55-10.bpo-34602.ZjHsYJ.rst b/Misc/NEWS.d/next/macOS/2021-08-27-16-55-10.bpo-34602.ZjHsYJ.rst deleted file mode 100644 index 29a6ff92554e1..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-08-27-16-55-10.bpo-34602.ZjHsYJ.rst +++ /dev/null @@ -1,3 +0,0 @@ -When building CPython on macOS with ``./configure ---with-undefined-behavior-sanitizer --with-pydebug``, the stack size is now -quadrupled to allow for the entire test suite to pass. diff --git a/Misc/NEWS.d/next/macOS/2021-08-30-00-04-10.bpo-45007.pixqUB.rst b/Misc/NEWS.d/next/macOS/2021-08-30-00-04-10.bpo-45007.pixqUB.rst deleted file mode 100644 index e4f1ac6de313d..0000000000000 --- a/Misc/NEWS.d/next/macOS/2021-08-30-00-04-10.bpo-45007.pixqUB.rst +++ /dev/null @@ -1 +0,0 @@ -Update macOS installer builds to use OpenSSL 1.1.1l. diff --git a/README.rst b/README.rst index 067d7d8ca2b81..c2b26a4651511 100644 --- a/README.rst +++ b/README.rst @@ -1,4 +1,4 @@ -This is Python version 3.11.0 alpha 0 +This is Python version 3.11.0 alpha 1 ===================================== .. image:: https://travis-ci.com/python/cpython.svg?branch=main From webhook-mailer at python.org Tue Oct 5 13:21:34 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 17:21:34 -0000 Subject: [Python-checkins] [doc] Fix gethostbyname_ex description (GH-28700) (GH-28742) Message-ID: https://github.com/python/cpython/commit/eb59e2fc111189804be307998a618f186b73e5fa commit: eb59e2fc111189804be307998a618f186b73e5fa branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-05T19:21:25+02:00 summary: [doc] Fix gethostbyname_ex description (GH-28700) (GH-28742) It seems part of `gethostbyname_ex` doc was copied from `gethostbyaddr`. The latter has an `ip_address` parameter whereas the former doesn't. (cherry picked from commit 4103280b83e1419bef535a42813d6dbe83bfe880) Co-authored-by: Andre Delfino files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 3c81858868010..b16bf22348fdf 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -827,8 +827,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: gethostbyname_ex(hostname) Translate a host name to IPv4 address format, extended interface. Return a - triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the primary - host name responding to the given *ip_address*, *aliaslist* is a (possibly + triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the host's + primary host name, *aliaslist* is a (possibly empty) list of alternative host names for the same address, and *ipaddrlist* is a list of IPv4 addresses for the same interface on the same host (often but not always a single address). :func:`gethostbyname_ex` does not support IPv6 name From webhook-mailer at python.org Tue Oct 5 13:22:21 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 17:22:21 -0000 Subject: [Python-checkins] [doc] Fix gethostbyname_ex description (GH-28700) (GH-28743) Message-ID: https://github.com/python/cpython/commit/950b324973a98ec45c21e7e7149415259a045ff7 commit: 950b324973a98ec45c21e7e7149415259a045ff7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-05T19:22:14+02:00 summary: [doc] Fix gethostbyname_ex description (GH-28700) (GH-28743) It seems part of `gethostbyname_ex` doc was copied from `gethostbyaddr`. The latter has an `ip_address` parameter whereas the former doesn't. (cherry picked from commit 4103280b83e1419bef535a42813d6dbe83bfe880) Co-authored-by: Andre Delfino Co-authored-by: Andre Delfino files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index cc8c5650dec3d..bc8723a90daad 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -814,8 +814,8 @@ The :mod:`socket` module also offers various network-related services: .. function:: gethostbyname_ex(hostname) Translate a host name to IPv4 address format, extended interface. Return a - triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the primary - host name responding to the given *ip_address*, *aliaslist* is a (possibly + triple ``(hostname, aliaslist, ipaddrlist)`` where *hostname* is the host's + primary host name, *aliaslist* is a (possibly empty) list of alternative host names for the same address, and *ipaddrlist* is a list of IPv4 addresses for the same interface on the same host (often but not always a single address). :func:`gethostbyname_ex` does not support IPv6 name From webhook-mailer at python.org Tue Oct 5 13:26:42 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Tue, 05 Oct 2021 17:26:42 -0000 Subject: [Python-checkins] bpo-45020: Identify which frozen modules are actually aliases. (gh-28655) Message-ID: https://github.com/python/cpython/commit/08285d563e64c179a56ab2f952345b3dbcdb54f3 commit: 08285d563e64c179a56ab2f952345b3dbcdb54f3 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-05T11:26:37-06:00 summary: bpo-45020: Identify which frozen modules are actually aliases. (gh-28655) In the list of generated frozen modules at the top of Tools/scripts/freeze_modules.py, you will find that some of the modules have a different name than the module (or .py file) that is actually frozen. Let's call each case an "alias". Aliases do not come into play until we get to the (generated) list of modules in Python/frozen.c. (The tool for freezing modules, Programs/_freeze_module, is only concerned with the source file, not the module it will be used for.) Knowledge of which frozen modules are aliases (and the identity of the original module) normally isn't important. However, this information is valuable when we go to set __file__ on frozen stdlib modules. This change updates Tools/scripts/freeze_modules.py to map aliases to the original module name (or None if not a stdlib module) in Python/frozen.c. We also add a helper function in Python/import.c to look up a frozen module's alias and add the result of that function to the frozen info returned from find_frozen(). https://bugs.python.org/issue45020 files: A Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-06-54.bpo-45020.Cj5VQN.rst M Include/internal/pycore_import.h M Lib/importlib/_bootstrap.py M Lib/test/test_importlib/frozen/test_finder.py M Lib/test/test_importlib/frozen/test_loader.py M Programs/_freeze_module.c M Python/clinic/import.c.h M Python/frozen.c M Python/import.c M Tools/scripts/freeze_modules.py diff --git a/Include/internal/pycore_import.h b/Include/internal/pycore_import.h index e21ed0a7a06a2..6439b7369fb59 100644 --- a/Include/internal/pycore_import.h +++ b/Include/internal/pycore_import.h @@ -10,6 +10,13 @@ extern PyStatus _PyImport_ReInitLock(void); #endif extern PyObject* _PyImport_BootstrapImp(PyThreadState *tstate); +struct _module_alias { + const char *name; /* ASCII encoded string */ + const char *orig; /* ASCII encoded string */ +}; + +extern const struct _module_alias * _PyImport_FrozenAliases; + #ifdef __cplusplus } #endif diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index 5807577c74bce..49f08814e9518 100644 --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -824,16 +824,39 @@ def module_repr(m): "slated for removal in Python 3.12", DeprecationWarning) return ''.format(m.__name__, FrozenImporter._ORIGIN) + @classmethod + def _setup_module(cls, module): + assert not hasattr(module, '__file__'), module.__file__ + ispkg = hasattr(module, '__path__') + assert not ispkg or not module.__path__, module.__path__ + spec = module.__spec__ + assert not ispkg or not spec.submodule_search_locations + + if spec.loader_state is None: + spec.loader_state = type(sys.implementation)( + data=None, + origname=None, + ) + elif not hasattr(spec.loader_state, 'data'): + spec.loader_state.data = None + if not getattr(spec.loader_state, 'origname', None): + origname = vars(module).pop('__origname__', None) + assert origname, 'see PyImport_ImportFrozenModuleObject()' + spec.loader_state.origname = origname + @classmethod def find_spec(cls, fullname, path=None, target=None): info = _call_with_frames_removed(_imp.find_frozen, fullname) if info is None: return None - data, ispkg = info + data, ispkg, origname = info spec = spec_from_loader(fullname, cls, origin=cls._ORIGIN, is_package=ispkg) - spec.loader_state = data + spec.loader_state = type(sys.implementation)( + data=data, + origname=origname, + ) return spec @classmethod @@ -857,7 +880,7 @@ def exec_module(module): spec = module.__spec__ name = spec.name try: - data = spec.loader_state + data = spec.loader_state.data except AttributeError: if not _imp.is_frozen(name): raise ImportError('{!r} is not a frozen module'.format(name), @@ -868,7 +891,7 @@ def exec_module(module): # Note that if this method is called again (e.g. by # importlib.reload()) then _imp.get_frozen_object() will notice # no data was provided and will look it up. - spec.loader_state = None + spec.loader_state.data = None code = _call_with_frames_removed(_imp.get_frozen_object, name, data) exec(code, module.__dict__) @@ -1220,6 +1243,8 @@ def _setup(sys_module, _imp_module): continue spec = _spec_from_module(module, loader) _init_module_attrs(spec, module) + if loader is FrozenImporter: + loader._setup_module(module) # Directly load built-in modules needed during bootstrap. self_module = sys.modules[__name__] diff --git a/Lib/test/test_importlib/frozen/test_finder.py b/Lib/test/test_importlib/frozen/test_finder.py index 0b15aeb598dd9..cd5586d524ce2 100644 --- a/Lib/test/test_importlib/frozen/test_finder.py +++ b/Lib/test/test_importlib/frozen/test_finder.py @@ -9,7 +9,15 @@ import unittest import warnings -from test.support import import_helper, REPO_ROOT +from test.support import import_helper, REPO_ROOT, STDLIB_DIR + + +def resolve_stdlib_file(name, ispkg=False): + assert name + if ispkg: + return os.path.join(STDLIB_DIR, *name.split('.'), '__init__.py') + else: + return os.path.join(STDLIB_DIR, *name.split('.')) + '.py' class FindSpecTests(abc.FinderTests): @@ -32,16 +40,30 @@ def check_basic(self, spec, name, ispkg=False): self.assertIsNone(spec.submodule_search_locations) self.assertIsNotNone(spec.loader_state) - def check_data(self, spec): + def check_loader_state(self, spec, origname=None, filename=None): + if not filename: + if not origname: + origname = spec.name + + actual = dict(vars(spec.loader_state)) + + # Check the code object used to import the frozen module. + # We can't compare the marshaled data directly because + # marshal.dumps() would mark "expected" (below) as a ref, + # which slightly changes the output. + # (See https://bugs.python.org/issue34093.) + data = actual.pop('data') with import_helper.frozen_modules(): expected = _imp.get_frozen_object(spec.name) - data = spec.loader_state - # We can't compare the marshaled data directly because - # marshal.dumps() would mark "expected" as a ref, which slightly - # changes the output. (See https://bugs.python.org/issue34093.) code = marshal.loads(data) self.assertEqual(code, expected) + # Check the rest of spec.loader_state. + expected = dict( + origname=origname, + ) + self.assertDictEqual(actual, expected) + def check_search_locations(self, spec): # Frozen packages do not have any path entries. # (See https://bugs.python.org/issue21736.) @@ -58,7 +80,7 @@ def test_module(self): with self.subTest(f'{name} -> {name}'): spec = self.find(name) self.check_basic(spec, name) - self.check_data(spec) + self.check_loader_state(spec) modules = { '__hello_alias__': '__hello__', '_frozen_importlib': 'importlib._bootstrap', @@ -67,26 +89,28 @@ def test_module(self): with self.subTest(f'{name} -> {origname}'): spec = self.find(name) self.check_basic(spec, name) - self.check_data(spec) + self.check_loader_state(spec, origname) modules = [ '__phello__.__init__', '__phello__.ham.__init__', ] for name in modules: - origname = name.rpartition('.')[0] + origname = '<' + name.rpartition('.')[0] + filename = resolve_stdlib_file(name) with self.subTest(f'{name} -> {origname}'): spec = self.find(name) self.check_basic(spec, name) - self.check_data(spec) + self.check_loader_state(spec, origname, filename) modules = { '__hello_only__': ('Tools', 'freeze', 'flag.py'), } for name, path in modules.items(): + origname = None filename = os.path.join(REPO_ROOT, *path) with self.subTest(f'{name} -> {filename}'): spec = self.find(name) self.check_basic(spec, name) - self.check_data(spec) + self.check_loader_state(spec, origname, filename) def test_package(self): packages = [ @@ -94,19 +118,21 @@ def test_package(self): '__phello__.ham', ] for name in packages: + filename = resolve_stdlib_file(name, ispkg=True) with self.subTest(f'{name} -> {name}'): spec = self.find(name) self.check_basic(spec, name, ispkg=True) - self.check_data(spec) + self.check_loader_state(spec, name, filename) self.check_search_locations(spec) packages = { '__phello_alias__': '__hello__', } for name, origname in packages.items(): + filename = resolve_stdlib_file(origname, ispkg=False) with self.subTest(f'{name} -> {origname}'): spec = self.find(name) self.check_basic(spec, name, ispkg=True) - self.check_data(spec) + self.check_loader_state(spec, origname, filename) self.check_search_locations(spec) # These are covered by test_module() and test_package(). diff --git a/Lib/test/test_importlib/frozen/test_loader.py b/Lib/test/test_importlib/frozen/test_loader.py index 992dcef05bcb1..d6f39fa98a6f1 100644 --- a/Lib/test/test_importlib/frozen/test_loader.py +++ b/Lib/test/test_importlib/frozen/test_loader.py @@ -32,17 +32,19 @@ def fresh(name, *, oldapi=False): class ExecModuleTests(abc.LoaderTests): - def exec_module(self, name): + def exec_module(self, name, origname=None): with import_helper.frozen_modules(): is_package = self.machinery.FrozenImporter.is_package(name) code = _imp.get_frozen_object(name) - data = marshal.dumps(code) spec = self.machinery.ModuleSpec( name, self.machinery.FrozenImporter, origin='frozen', is_package=is_package, - loader_state=data, + loader_state=types.SimpleNamespace( + data=marshal.dumps(code), + origname=origname or name, + ), ) module = types.ModuleType(name) module.__spec__ = spec @@ -66,7 +68,8 @@ def test_module(self): self.assertEqual(getattr(module, attr), value) self.assertEqual(output, 'Hello world!\n') self.assertTrue(hasattr(module, '__spec__')) - self.assertIsNone(module.__spec__.loader_state) + self.assertIsNone(module.__spec__.loader_state.data) + self.assertEqual(module.__spec__.loader_state.origname, name) def test_package(self): name = '__phello__' @@ -79,7 +82,8 @@ def test_package(self): name=name, attr=attr, given=attr_value, expected=value)) self.assertEqual(output, 'Hello world!\n') - self.assertIsNone(module.__spec__.loader_state) + self.assertIsNone(module.__spec__.loader_state.data) + self.assertEqual(module.__spec__.loader_state.origname, name) def test_lacking_parent(self): name = '__phello__.spam' diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-06-54.bpo-45020.Cj5VQN.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-06-54.bpo-45020.Cj5VQN.rst new file mode 100644 index 0000000000000..839604357d121 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-06-54.bpo-45020.Cj5VQN.rst @@ -0,0 +1,5 @@ +For frozen stdlib modules, record the original module name as +``module.__spec__.loader_state.origname``. If the value is different than +``module.__spec__.name`` then the module was defined as an alias in +Tools/scripts/freeze_modules.py. If it is ``None`` then the module comes +from a source file outside the stdlib. diff --git a/Programs/_freeze_module.c b/Programs/_freeze_module.c index 4b0633e7e7a3d..dd90d92e512c0 100644 --- a/Programs/_freeze_module.c +++ b/Programs/_freeze_module.c @@ -9,6 +9,7 @@ #include #include +#include #include #include @@ -24,8 +25,12 @@ static const struct _frozen _PyImport_FrozenModules[] = { {0, 0, 0} /* sentinel */ }; +static const struct _module_alias aliases[] = { + {0, 0} /* sentinel */ +}; const struct _frozen *PyImport_FrozenModules; +const struct _module_alias *_PyImport_FrozenAliases; static const char header[] = "/* Auto-generated by Programs/_freeze_module.c */"; @@ -183,6 +188,7 @@ main(int argc, char *argv[]) const char *name, *inpath, *outpath; PyImport_FrozenModules = _PyImport_FrozenModules; + _PyImport_FrozenAliases = aliases; if (argc != 4) { fprintf(stderr, "need to specify the name, input and output paths\n"); diff --git a/Python/clinic/import.c.h b/Python/clinic/import.c.h index 09738834195c7..dfb59de3b5ce8 100644 --- a/Python/clinic/import.c.h +++ b/Python/clinic/import.c.h @@ -178,7 +178,10 @@ PyDoc_STRVAR(_imp_find_frozen__doc__, "The returned info (a 2-tuple):\n" "\n" " * data the raw marshalled bytes\n" -" * is_package whether or not it is a package"); +" * is_package whether or not it is a package\n" +" * origname the originally frozen module\'s name, or None if not\n" +" a stdlib module (this will usually be the same as\n" +" the module\'s current name)"); #define _IMP_FIND_FROZEN_METHODDEF \ {"find_frozen", (PyCFunction)_imp_find_frozen, METH_O, _imp_find_frozen__doc__}, @@ -545,4 +548,4 @@ _imp_source_hash(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyOb #ifndef _IMP_EXEC_DYNAMIC_METHODDEF #define _IMP_EXEC_DYNAMIC_METHODDEF #endif /* !defined(_IMP_EXEC_DYNAMIC_METHODDEF) */ -/*[clinic end generated code: output=a31e1c00653359ff input=a9049054013a1b77]*/ +/*[clinic end generated code: output=8c8dd08158f9ac7c input=a9049054013a1b77]*/ diff --git a/Python/frozen.c b/Python/frozen.c index b4f7121fda35f..499b3b9957057 100644 --- a/Python/frozen.c +++ b/Python/frozen.c @@ -36,6 +36,7 @@ and __phello__.spam. Loading any will print some famous words... */ #include "Python.h" +#include "pycore_import.h" /* Includes for frozen modules: */ #include "frozen_modules/importlib._bootstrap.h" @@ -102,9 +103,24 @@ static const struct _frozen _PyImport_FrozenModules[] = { {"__phello__.spam", _Py_M____phello___spam, (int)sizeof(_Py_M____phello___spam)}, {"__hello_only__", _Py_M__frozen_only, (int)sizeof(_Py_M__frozen_only)}, - {0, 0, 0} /* sentinel */ + {0, 0, 0} /* modules sentinel */ }; +static const struct _module_alias aliases[] = { + {"_frozen_importlib", "importlib._bootstrap"}, + {"_frozen_importlib_external", "importlib._bootstrap_external"}, + {"os.path", "posixpath"}, + {"__hello_alias__", "__hello__"}, + {"__phello_alias__", "__hello__"}, + {"__phello_alias__.spam", "__hello__"}, + {"__phello__.__init__", "<__phello__"}, + {"__phello__.ham.__init__", "<__phello__.ham"}, + {"__hello_only__", NULL}, + {0, 0} /* aliases sentinel */ +}; +const struct _module_alias *_PyImport_FrozenAliases = aliases; + + /* Embedding apps may change this pointer to point to their favorite collection of frozen modules: */ diff --git a/Python/import.c b/Python/import.c index 22cefdf08b48f..a6170a39c7fdb 100644 --- a/Python/import.c +++ b/Python/import.c @@ -1046,6 +1046,29 @@ _imp_create_builtin(PyObject *module, PyObject *spec) } +/* Return true if the name is an alias. In that case, "alias" is set + to the original module name. If it is an alias but the original + module isn't known then "alias" is set to NULL while true is returned. */ +static bool +resolve_module_alias(const char *name, const struct _module_alias *aliases, + const char **alias) +{ + const struct _module_alias *entry; + for (entry = aliases; ; entry++) { + if (entry->name == NULL) { + /* It isn't an alias. */ + return false; + } + if (strcmp(name, entry->name) == 0) { + if (alias != NULL) { + *alias = entry->orig; + } + return true; + } + } +} + + /* Frozen modules */ static bool @@ -1161,16 +1184,15 @@ struct frozen_info { const char *data; Py_ssize_t size; bool is_package; + bool is_alias; + const char *origname; }; static frozen_status find_frozen(PyObject *nameobj, struct frozen_info *info) { if (info != NULL) { - info->nameobj = NULL; - info->data = NULL; - info->size = 0; - info->is_package = false; + memset(info, 0, sizeof(*info)); } if (nameobj == NULL || nameobj == Py_None) { @@ -1205,6 +1227,9 @@ find_frozen(PyObject *nameobj, struct frozen_info *info) info->data = (const char *)p->code; info->size = p->size < 0 ? -(p->size) : p->size; info->is_package = p->size < 0 ? true : false; + info->origname = name; + info->is_alias = resolve_module_alias(name, _PyImport_FrozenAliases, + &info->origname); } if (p->code == NULL) { @@ -1246,7 +1271,8 @@ int PyImport_ImportFrozenModuleObject(PyObject *name) { PyThreadState *tstate = _PyThreadState_GET(); - PyObject *co, *m, *d; + PyObject *co, *m, *d = NULL; + int err; struct frozen_info info; frozen_status status = find_frozen(name, &info); @@ -1267,7 +1293,6 @@ PyImport_ImportFrozenModuleObject(PyObject *name) if (info.is_package) { /* Set __path__ to the empty list */ PyObject *l; - int err; m = import_add_module(tstate, name); if (m == NULL) goto err_return; @@ -1288,15 +1313,33 @@ PyImport_ImportFrozenModuleObject(PyObject *name) goto err_return; } m = exec_code_in_module(tstate, name, d, co); - Py_DECREF(d); if (m == NULL) { goto err_return; } - Py_DECREF(co); Py_DECREF(m); + /* Set __origname__ (consumed in FrozenImporter._setup_module()). */ + PyObject *origname; + if (info.origname) { + origname = PyUnicode_FromString(info.origname); + if (origname == NULL) { + goto err_return; + } + } + else { + Py_INCREF(Py_None); + origname = Py_None; + } + err = PyDict_SetItemString(d, "__origname__", origname); + Py_DECREF(origname); + if (err != 0) { + goto err_return; + } + Py_DECREF(d); + Py_DECREF(co); return 1; err_return: + Py_XDECREF(d); Py_DECREF(co); return -1; } @@ -2014,11 +2057,14 @@ The returned info (a 2-tuple): * data the raw marshalled bytes * is_package whether or not it is a package + * origname the originally frozen module's name, or None if not + a stdlib module (this will usually be the same as + the module's current name) [clinic start generated code]*/ static PyObject * _imp_find_frozen_impl(PyObject *module, PyObject *name) -/*[clinic end generated code: output=3fd17da90d417e4e input=4e52b3ac95f6d7ab]*/ +/*[clinic end generated code: output=3fd17da90d417e4e input=6aa7b9078a89280a]*/ { struct frozen_info info; frozen_status status = find_frozen(name, &info); @@ -2032,12 +2078,25 @@ _imp_find_frozen_impl(PyObject *module, PyObject *name) set_frozen_error(status, name); return NULL; } + PyObject *data = PyBytes_FromStringAndSize(info.data, info.size); if (data == NULL) { return NULL; } - PyObject *result = PyTuple_Pack(2, data, - info.is_package ? Py_True : Py_False); + + PyObject *origname = NULL; + if (info.origname != NULL && info.origname[0] != '\0') { + origname = PyUnicode_FromString(info.origname); + if (origname == NULL) { + Py_DECREF(data); + return NULL; + } + } + + PyObject *result = PyTuple_Pack(3, data, + info.is_package ? Py_True : Py_False, + origname ? origname : Py_None); + Py_XDECREF(origname); Py_DECREF(data); return result; } diff --git a/Tools/scripts/freeze_modules.py b/Tools/scripts/freeze_modules.py index 6091d831a8d31..36e284100ed52 100644 --- a/Tools/scripts/freeze_modules.py +++ b/Tools/scripts/freeze_modules.py @@ -274,6 +274,15 @@ def symbol(self): name = self.frozenid.replace('.', '_') return '_Py_M__' + name + @property + def ispkg(self): + if not self.pyfile: + return False + elif self.frozenid.endswith('.__init__'): + return False + else: + return os.path.basename(self.pyfile) == '__init__.py' + def resolve_frozen_file(frozenid, destdir=MODULES_DIR): """Return the filename corresponding to the given frozen ID. @@ -305,6 +314,17 @@ def __getattr__(self, name): def modname(self): return self.name + @property + def orig(self): + return self.source.modname + + @property + def isalias(self): + orig = self.source.modname + if not orig: + return True + return self.name != orig + def summarize(self): source = self.source.modname if source: @@ -507,6 +527,7 @@ def regen_frozen(modules): headerlines.append(f'#include "{header}"') deflines = [] + aliaslines = [] indent = ' ' lastsection = None for mod in modules: @@ -528,6 +549,15 @@ def regen_frozen(modules): deflines.append(line1) deflines.append(indent + line2) + if mod.isalias: + if not mod.orig: + entry = '{"%s", NULL},' % (mod.name,) + elif mod.source.ispkg: + entry = '{"%s", "<%s"},' % (mod.name, mod.orig) + else: + entry = '{"%s", "%s"},' % (mod.name, mod.orig) + aliaslines.append(indent + entry) + if not deflines[0]: del deflines[0] for i, line in enumerate(deflines): @@ -549,10 +579,17 @@ def regen_frozen(modules): lines = replace_block( lines, "static const struct _frozen _PyImport_FrozenModules[] =", - "/* sentinel */", + "/* modules sentinel */", deflines, FROZEN_FILE, ) + lines = replace_block( + lines, + "const struct _module_alias aliases[] =", + "/* aliases sentinel */", + aliaslines, + FROZEN_FILE, + ) outfile.writelines(lines) From webhook-mailer at python.org Tue Oct 5 16:30:43 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 20:30:43 -0000 Subject: [Python-checkins] [3.9] bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) (GH-28741) Message-ID: https://github.com/python/cpython/commit/52d9d3b75441ae6038fadead89eac5eecdd34501 commit: 52d9d3b75441ae6038fadead89eac5eecdd34501 branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-05T22:30:25+02:00 summary: [3.9] bpo-44050: Extension modules can share state when they don't support sub-interpreters. (GH-27794) (GH-28741) (cherry picked from commit b9bb74871b27d9226df2dd3fce9d42bda8b43c2b) Co-authored-by: Hai Shi files: A Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst M Lib/test/test_capi.py M Modules/_testmultiphase.c M Python/import.c diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index 0cbe839665e3f..6b2fe2f1580c3 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -680,6 +680,37 @@ def test_mutate_exception(self): self.assertFalse(hasattr(binascii.Error, "foobar")) + def test_module_state_shared_in_global(self): + """ + bpo-44050: Extension module state should be shared between interpreters + when it doesn't support sub-interpreters. + """ + r, w = os.pipe() + self.addCleanup(os.close, r) + self.addCleanup(os.close, w) + + script = textwrap.dedent(f""" + import importlib.machinery + import importlib.util + import os + + fullname = '_test_module_state_shared' + origin = importlib.util.find_spec('_testmultiphase').origin + loader = importlib.machinery.ExtensionFileLoader(fullname, origin) + spec = importlib.util.spec_from_loader(fullname, loader) + module = importlib.util.module_from_spec(spec) + attr_id = str(id(module.Error)).encode() + + os.write({w}, attr_id) + """) + exec(script) + main_attr_id = os.read(r, 100) + + ret = support.run_in_subinterp(script) + self.assertEqual(ret, 0) + subinterp_attr_id = os.read(r, 100) + self.assertEqual(main_attr_id, subinterp_attr_id) + class TestThreadState(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst new file mode 100644 index 0000000000000..d6eed9f1bcfe9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-00-30-09.bpo-44050.mFI15u.rst @@ -0,0 +1,3 @@ +Extensions that indicate they use global state (by setting ``m_size`` to -1) +can again be used in multiple interpreters. This reverts to behavior of +Python 3.8. diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index d69ae628fa7a4..4a4dbe77acfac 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -834,6 +834,30 @@ PyInit__testmultiphase_meth_state_access(PyObject *spec) return PyModuleDef_Init(&def_meth_state_access); } +static PyModuleDef def_module_state_shared = { + PyModuleDef_HEAD_INIT, + .m_name = "_test_module_state_shared", + .m_doc = PyDoc_STR("Regression Test module for single-phase init."), + .m_size = -1, +}; + +PyMODINIT_FUNC +PyInit__test_module_state_shared(PyObject *spec) +{ + PyObject *module = PyModule_Create(&def_module_state_shared); + if (module == NULL) { + return NULL; + } + + Py_INCREF(PyExc_Exception); + if (PyModule_AddObject(module, "Error", PyExc_Exception) < 0) { + Py_DECREF(PyExc_Exception); + Py_DECREF(module); + return NULL; + } + return module; +} + /*** Helper for imp test ***/ diff --git a/Python/import.c b/Python/import.c index 1ec755393df99..8358d70f468dd 100644 --- a/Python/import.c +++ b/Python/import.c @@ -710,7 +710,9 @@ _PyImport_FixupExtensionObject(PyObject *mod, PyObject *name, return -1; } - if (_Py_IsMainInterpreter(tstate)) { + // bpo-44050: Extensions and def->m_base.m_copy can be updated + // when the extension module doesn't support sub-interpreters. + if (_Py_IsMainInterpreter(tstate) || def->m_size == -1) { if (def->m_size == -1) { if (def->m_base.m_copy) { /* Somebody already imported the module, From webhook-mailer at python.org Tue Oct 5 17:04:18 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 21:04:18 -0000 Subject: [Python-checkins] sqlite3: Modernize documentation around unicode and bytes. (GH-28652) (GH-28695) Message-ID: https://github.com/python/cpython/commit/92450f23c6b2609ed40410c84cd4ed4e7ba72d4a commit: 92450f23c6b2609ed40410c84cd4ed4e7ba72d4a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-05T23:04:02+02:00 summary: sqlite3: Modernize documentation around unicode and bytes. (GH-28652) (GH-28695) (cherry picked from commit 1dac95c814763eb8a53896ac4326d8d51895d43d) Co-authored-by: Julien Palard files: M Doc/includes/sqlite3/text_factory.py M Doc/library/sqlite3.rst diff --git a/Doc/includes/sqlite3/text_factory.py b/Doc/includes/sqlite3/text_factory.py index a857a155cdd4f..c0d87cd559118 100644 --- a/Doc/includes/sqlite3/text_factory.py +++ b/Doc/includes/sqlite3/text_factory.py @@ -3,9 +3,9 @@ con = sqlite3.connect(":memory:") cur = con.cursor() -AUSTRIA = "\xd6sterreich" +AUSTRIA = "?sterreich" -# by default, rows are returned as Unicode +# by default, rows are returned as str cur.execute("select ?", (AUSTRIA,)) row = cur.fetchone() assert row[0] == AUSTRIA diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 4936df5a27e2c..aeedcbeffcf4e 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -513,8 +513,8 @@ Connection Objects Using this attribute you can control what objects are returned for the ``TEXT`` data type. By default, this attribute is set to :class:`str` and the - :mod:`sqlite3` module will return Unicode objects for ``TEXT``. If you want to - return bytestrings instead, you can set it to :class:`bytes`. + :mod:`sqlite3` module will return :class:`str` objects for ``TEXT``. + If you want to return :class:`bytes` instead, you can set it to :class:`bytes`. You can also set it to any other callable that accepts a single bytestring parameter and returns the resulting object. From webhook-mailer at python.org Tue Oct 5 17:04:32 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 21:04:32 -0000 Subject: [Python-checkins] sqlite3: Modernize documentation around unicode and bytes. (GH-28652) (GH-28694) Message-ID: https://github.com/python/cpython/commit/258c5fbbfd5b09d5ece629d7877d8753112f0f1c commit: 258c5fbbfd5b09d5ece629d7877d8753112f0f1c branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-05T23:04:27+02:00 summary: sqlite3: Modernize documentation around unicode and bytes. (GH-28652) (GH-28694) (cherry picked from commit 1dac95c814763eb8a53896ac4326d8d51895d43d) Co-authored-by: Julien Palard files: M Doc/includes/sqlite3/text_factory.py M Doc/library/sqlite3.rst diff --git a/Doc/includes/sqlite3/text_factory.py b/Doc/includes/sqlite3/text_factory.py index a857a155cdd4f..c0d87cd559118 100644 --- a/Doc/includes/sqlite3/text_factory.py +++ b/Doc/includes/sqlite3/text_factory.py @@ -3,9 +3,9 @@ con = sqlite3.connect(":memory:") cur = con.cursor() -AUSTRIA = "\xd6sterreich" +AUSTRIA = "?sterreich" -# by default, rows are returned as Unicode +# by default, rows are returned as str cur.execute("select ?", (AUSTRIA,)) row = cur.fetchone() assert row[0] == AUSTRIA diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index d29e425cb9078..5641a347ceda1 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -528,8 +528,8 @@ Connection Objects Using this attribute you can control what objects are returned for the ``TEXT`` data type. By default, this attribute is set to :class:`str` and the - :mod:`sqlite3` module will return Unicode objects for ``TEXT``. If you want to - return bytestrings instead, you can set it to :class:`bytes`. + :mod:`sqlite3` module will return :class:`str` objects for ``TEXT``. + If you want to return :class:`bytes` instead, you can set it to :class:`bytes`. You can also set it to any other callable that accepts a single bytestring parameter and returns the resulting object. From webhook-mailer at python.org Tue Oct 5 17:06:25 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 21:06:25 -0000 Subject: [Python-checkins] [Tools/peg_generator/pegen/parser.py] Fix typo: s/wether/whether/ (GH-28739) Message-ID: https://github.com/python/cpython/commit/48fadb1f19f75fcaacb7d523dcc9e05b97dcce74 commit: 48fadb1f19f75fcaacb7d523dcc9e05b97dcce74 branch: main author: Ikko Ashimine committer: ambv date: 2021-10-05T23:06:19+02:00 summary: [Tools/peg_generator/pegen/parser.py] Fix typo: s/wether/whether/ (GH-28739) files: M Tools/peg_generator/pegen/parser.py diff --git a/Tools/peg_generator/pegen/parser.py b/Tools/peg_generator/pegen/parser.py index 4ce60e35dd0e0..034e8e6017a22 100644 --- a/Tools/peg_generator/pegen/parser.py +++ b/Tools/peg_generator/pegen/parser.py @@ -168,7 +168,7 @@ def __init__(self, tokenizer: Tokenizer, *, verbose: bool = False): self._verbose = verbose self._level = 0 self._cache: Dict[Tuple[Mark, str, Tuple[Any, ...]], Tuple[Any, Mark]] = {} - # Integer tracking wether we are in a left recursive rule or not. Can be useful + # Integer tracking whether we are in a left recursive rule or not. Can be useful # for error reporting. self.in_recursive_rule = 0 # Pass through common tokenizer methods. From webhook-mailer at python.org Tue Oct 5 17:30:49 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 21:30:49 -0000 Subject: [Python-checkins] bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) Message-ID: https://github.com/python/cpython/commit/4c8d543823dde5a30615da61727837a48f7ab847 commit: 4c8d543823dde5a30615da61727837a48f7ab847 branch: main author: Illia Volochii committer: ambv date: 2021-10-05T23:30:38+02:00 summary: bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) files: A Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl A Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl A Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst D Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl D Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl M Lib/ensurepip/__init__.py diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index 651f876e24bc8..94f1b6604cb7a 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -10,8 +10,8 @@ __all__ = ["version", "bootstrap"] _PACKAGE_NAMES = ('setuptools', 'pip') -_SETUPTOOLS_VERSION = "57.4.0" -_PIP_VERSION = "21.2.3" +_SETUPTOOLS_VERSION = "58.1.0" +_PIP_VERSION = "21.2.4" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION, "py3"), ("pip", _PIP_VERSION, "py3"), @@ -41,7 +41,7 @@ def _find_packages(path): # comparison since this case should not happen. filenames = sorted(filenames) for filename in filenames: - # filename is like 'pip-20.2.3-py2.py3-none-any.whl' + # filename is like 'pip-21.2.4-py3-none-any.whl' if not filename.endswith(".whl"): continue for name in _PACKAGE_NAMES: @@ -51,7 +51,7 @@ def _find_packages(path): else: continue - # Extract '20.2.2' from 'pip-20.2.2-py2.py3-none-any.whl' + # Extract '21.2.4' from 'pip-21.2.4-py3-none-any.whl' version = filename.removeprefix(prefix).partition('-')[0] wheel_path = os.path.join(path, filename) packages[name] = _Package(version, None, wheel_path) diff --git a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl deleted file mode 100644 index d417df63d9ec1..0000000000000 Binary files a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl and /dev/null differ diff --git a/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl new file mode 100644 index 0000000000000..46d3012c59b17 Binary files /dev/null and b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl differ diff --git a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl similarity index 80% rename from Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl rename to Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl index af8f8ba21003b..18c8c22958f1f 100644 Binary files a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl and b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl differ diff --git a/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst new file mode 100644 index 0000000000000..8dac4e6a25271 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst @@ -0,0 +1 @@ +Update bundled pip to 21.2.4 and setuptools to 58.1.0 From webhook-mailer at python.org Tue Oct 5 17:48:52 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 21:48:52 -0000 Subject: [Python-checkins] [doc] Fix typos found using codespell (GH-28744) Message-ID: https://github.com/python/cpython/commit/241bda785a092a272d7e0f6a4e20bd250c389cfe commit: 241bda785a092a272d7e0f6a4e20bd250c389cfe branch: main author: Christian Clauss committer: ambv date: 2021-10-05T23:48:44+02:00 summary: [doc] Fix typos found using codespell (GH-28744) Co-authored-by: ?ukasz Langa files: M Doc/c-api/call.rst M Doc/c-api/init_config.rst M Doc/faq/design.rst M Doc/library/ast.rst M Doc/library/base64.rst M Doc/library/dis.rst M Doc/library/enum.rst M Doc/library/fileinput.rst M Doc/library/importlib.metadata.rst M Doc/library/test.rst M Doc/library/types.rst M Doc/reference/import.rst M Doc/tools/rstlint.py M Doc/using/cmdline.rst M Doc/using/configure.rst M Doc/whatsnew/3.10.rst diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 31dc9c8031fdb..739b5e97d1515 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -185,7 +185,7 @@ Object Calling API Various functions are available for calling a Python object. Each converts its arguments to a convention supported by the called object ? either *tp_call* or vectorcall. -In order to do as litle conversion as possible, pick one that best fits +In order to do as little conversion as possible, pick one that best fits the format of data you have available. The following table summarizes the available functions; diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index 2e52679ebc5b5..989660caeda06 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -22,7 +22,7 @@ There are two kinds of configuration: * The :ref:`Isolated Configuration ` can be used to embed Python into an application. It isolates Python from the system. For example, environments variables are ignored, the LC_CTYPE locale is left unchanged and - no signal handler is registred. + no signal handler is registered. The :c:func:`Py_RunMain` function can be used to write a customized Python program. @@ -706,7 +706,7 @@ PyConfig * Otherwise, use the :term:`locale encoding`: ``nl_langinfo(CODESET)`` result. - At Python statup, the encoding name is normalized to the Python codec + At Python startup, the encoding name is normalized to the Python codec name. For example, ``"ANSI_X3.4-1968"`` is replaced with ``"ascii"``. See also the :c:member:`~PyConfig.filesystem_errors` member. diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 720b1e496eb84..0437b59d55da6 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -714,7 +714,7 @@ Why don't generators support the with statement? For technical reasons, a generator used directly as a context manager would not work correctly. When, as is most common, a generator is used as an iterator run to completion, no closing is needed. When it is, wrap -it as "contextlib.closing(generator)" in the 'with' statment. +it as "contextlib.closing(generator)" in the 'with' statement. Why are colons required for the if/while/def/class statements? diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index d84c841fa4a08..e29b5e88d71d4 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1920,7 +1920,7 @@ and classes for traversing abstract syntax trees: If source contains a null character ('\0'), :exc:`ValueError` is raised. .. warning:: - Note that succesfully parsing souce code into an AST object doesn't + Note that successfully parsing source code into an AST object doesn't guarantee that the source code provided is valid Python code that can be executed as the compilation step can raise further :exc:`SyntaxError` exceptions. For instance, the source ``return 42`` generates a valid diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index 29e52ad48e115..4ff038c8d29f1 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -154,7 +154,7 @@ The modern interface provides: This version does not allow the digit 0 (zero) to the letter O (oh) and digit 1 (one) to either the letter I (eye) or letter L (el) mappings, all these characters are included in the Extended Hex Alphabet and are not - interchangable. + interchangeable. .. versionadded:: 3.10 diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 9b683083f6b62..9958dea5587f1 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -725,7 +725,7 @@ iterations of the loop. of the stack and sets the ``f_lasti`` attribute of the frame with that value. Then pops the next exception from the stack uses it to restore the current exception. Finally it re-raises the originally popped exception. - Used in excpetion handler cleanup. + Used in exception handler cleanup. .. versionadded:: 3.11 diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index f28505a6d2ccf..86bf705af77e7 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -651,8 +651,8 @@ Data Types --------------- -Utilites and Decorators ------------------------ +Utilities and Decorators +------------------------ .. class:: auto diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index ae31341a9e63e..2a2c1b3c2b921 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -50,7 +50,7 @@ You can control how files are opened by providing an opening hook via the *openhook* parameter to :func:`fileinput.input` or :class:`FileInput()`. The hook must be a function that takes two arguments, *filename* and *mode*, and returns an accordingly opened file-like object. If *encoding* and/or *errors* -are specified, they will be passed to the hook as aditional keyword arguments. +are specified, they will be passed to the hook as additional keyword arguments. This module provides a :func:`hook_compressed` to support compressed files. The following function is the primary interface of this module: diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index c43457a385067..58c582d712413 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -255,7 +255,7 @@ function:: Package distributions --------------------- -A convience method to resolve the distribution or +A convenience method to resolve the distribution or distributions (in the case of a namespace package) for top-level Python packages or modules:: diff --git a/Doc/library/test.rst b/Doc/library/test.rst index a6cc2be4d5522..a8dc35476fc9b 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -1249,7 +1249,7 @@ The :mod:`test.support.threading_helper` module provides support for threading t Context manager catching :class:`threading.Thread` exception using :func:`threading.excepthook`. - Attributes set when an exception is catched: + Attributes set when an exception is caught: * ``exc_type`` * ``exc_value`` @@ -1458,7 +1458,7 @@ The :mod:`test.support.os_helper` module provides support for os tests. .. function:: unlink(filename) Call :func:`os.unlink` on *filename*. On Windows platforms, this is - wrapped with a wait loop that checks for the existence fo the file. + wrapped with a wait loop that checks for the existence of the file. :mod:`test.support.import_helper` --- Utilities for import tests diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 88fe47a8d4a7f..2314b02c7449c 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -243,7 +243,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__loader__", None)`` if you explicitly need to use this attribute. @@ -268,7 +268,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__package__", None)`` if you explicitly need to use this attribute. diff --git a/Doc/reference/import.rst b/Doc/reference/import.rst index 39fcba015b694..69e2a94727449 100644 --- a/Doc/reference/import.rst +++ b/Doc/reference/import.rst @@ -605,7 +605,7 @@ the module. ``__file__`` is optional (if set, value must be a string). It indicates the pathname of the file from which the module was loaded (if - loaded from a file), or the pathname of the shared libray file + loaded from a file), or the pathname of the shared library file for extension modules loaded dynamically from a shared library. It might be missing for certain types of modules, such as C modules that are statically linked into the interpreter, and the diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index 91aed80b1e9a6..045b7d7c945df 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -256,7 +256,7 @@ def hide_comments(lines): """Tool to remove comments from given lines. It yields empty lines in place of comments, so line numbers are - still meaningfull. + still meaningful. """ in_multiline_comment = False for line in lines: diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index b42518e8731ca..23a645a036aed 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -477,7 +477,7 @@ Miscellaneous options * ``-X no_debug_ranges`` disables the inclusion of the tables mapping extra location information (end line, start column offset and end column offset) to every instruction in code objects. This is useful when smaller code - objects and pyc files are desired as well as supressing the extra visual + objects and pyc files are desired as well as suppressing the extra visual location indicators when the interpreter displays tracebacks. See also :envvar:`PYTHONNODEBUGRANGES`. * ``-X frozen_modules`` determines whether or not frozen modules are @@ -959,7 +959,7 @@ conflict. If this variable is set, it disables the inclusion of the tables mapping extra location information (end line, start column offset and end column offset) to every instruction in code objects. This is useful when smaller - code objects and pyc files are desired as well as supressing the extra visual + code objects and pyc files are desired as well as suppressing the extra visual location indicators when the interpreter displays tracebacks. .. versionadded:: 3.11 diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index a545d5a9372ac..a59bee4ccf0f6 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -553,7 +553,7 @@ Built-in modules have no ``__file__`` attribute:: File "", line 1, in AttributeError: module 'sys' has no attribute '__file__' -Other C extensions are built as dynamic libraires, like the ``_asyncio`` module. +Other C extensions are built as dynamic libraries, like the ``_asyncio`` module. They are built with the ``Py_BUILD_CORE_MODULE`` macro defined. Example on Linux x86-64:: diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index de25d158bd54a..b063255bc3e9e 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1583,7 +1583,7 @@ Deprecated * Currently Python accepts numeric literals immediately followed by keywords, for example ``0in x``, ``1or x``, ``0if 1else 2``. It allows confusing - and ambigious expressions like ``[0x1for x in y]`` (which can be + and ambiguous expressions like ``[0x1for x in y]`` (which can be interpreted as ``[0x1 for x in y]`` or ``[0x1f or x in y]``). Starting in this release, a deprecation warning is raised if the numeric literal is immediately followed by one of keywords :keyword:`and`, :keyword:`else`, @@ -1920,7 +1920,7 @@ Changes in the Python API if the *globals* dictionary has no ``"__builtins__"`` key, rather than using ``{"None": None}`` as builtins: same behavior as :func:`eval` and :func:`exec` functions. Defining a function with ``def function(...): ...`` - in Python is not affected, globals cannot be overriden with this syntax: it + in Python is not affected, globals cannot be overridden with this syntax: it also inherits the current builtins. (Contributed by Victor Stinner in :issue:`42990`.) From webhook-mailer at python.org Tue Oct 5 18:15:51 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 22:15:51 -0000 Subject: [Python-checkins] bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) (GH-28746) Message-ID: https://github.com/python/cpython/commit/325e4647afffe347cc20747f3dccc6ba9e782636 commit: 325e4647afffe347cc20747f3dccc6ba9e782636 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T00:15:46+02:00 summary: bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) (GH-28746) (cherry picked from commit 4c8d543823dde5a30615da61727837a48f7ab847) Co-authored-by: Illia Volochii files: A Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl A Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl A Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst D Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl D Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl M Lib/ensurepip/__init__.py diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index f28ab11ed4008..0a5730c00c359 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -11,8 +11,8 @@ __all__ = ["version", "bootstrap"] _PACKAGE_NAMES = ('setuptools', 'pip') -_SETUPTOOLS_VERSION = "57.4.0" -_PIP_VERSION = "21.2.3" +_SETUPTOOLS_VERSION = "58.1.0" +_PIP_VERSION = "21.2.4" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION, "py3"), ("pip", _PIP_VERSION, "py3"), @@ -42,7 +42,7 @@ def _find_packages(path): # comparison since this case should not happen. filenames = sorted(filenames) for filename in filenames: - # filename is like 'pip-20.2.3-py2.py3-none-any.whl' + # filename is like 'pip-21.2.4-py3-none-any.whl' if not filename.endswith(".whl"): continue for name in _PACKAGE_NAMES: @@ -52,7 +52,7 @@ def _find_packages(path): else: continue - # Extract '20.2.2' from 'pip-20.2.2-py2.py3-none-any.whl' + # Extract '21.2.4' from 'pip-21.2.4-py3-none-any.whl' version = filename.removeprefix(prefix).partition('-')[0] wheel_path = os.path.join(path, filename) packages[name] = _Package(version, None, wheel_path) diff --git a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl deleted file mode 100644 index d417df63d9ec1..0000000000000 Binary files a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl and /dev/null differ diff --git a/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl new file mode 100644 index 0000000000000..46d3012c59b17 Binary files /dev/null and b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl differ diff --git a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl similarity index 80% rename from Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl rename to Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl index af8f8ba21003b..18c8c22958f1f 100644 Binary files a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl and b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl differ diff --git a/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst new file mode 100644 index 0000000000000..8dac4e6a25271 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst @@ -0,0 +1 @@ +Update bundled pip to 21.2.4 and setuptools to 58.1.0 From webhook-mailer at python.org Tue Oct 5 18:16:34 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 05 Oct 2021 22:16:34 -0000 Subject: [Python-checkins] [3.9] bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) (GH-28747) Message-ID: https://github.com/python/cpython/commit/1374459c9088725e022beab418bced96a825baad commit: 1374459c9088725e022beab418bced96a825baad branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-06T00:16:30+02:00 summary: [3.9] bpo-45343: Update bundled pip to 21.2.4 and setuptools to 58.1.0 (GH-28684) (GH-28747) (cherry picked from commit 4c8d543823dde5a30615da61727837a48f7ab847) Co-authored-by: Illia Volochii files: A Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl A Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl A Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst D Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl D Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl M Lib/ensurepip/__init__.py diff --git a/Lib/ensurepip/__init__.py b/Lib/ensurepip/__init__.py index 7b03970c93ec0..2a140a2624d46 100644 --- a/Lib/ensurepip/__init__.py +++ b/Lib/ensurepip/__init__.py @@ -11,12 +11,8 @@ __all__ = ["version", "bootstrap"] - - -_SETUPTOOLS_VERSION = "57.4.0" - -_PIP_VERSION = "21.2.3" - +_SETUPTOOLS_VERSION = "58.1.0" +_PIP_VERSION = "21.2.4" _PROJECTS = [ ("setuptools", _SETUPTOOLS_VERSION, "py3"), ("pip", _PIP_VERSION, "py3"), diff --git a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl deleted file mode 100644 index d417df63d9ec1..0000000000000 Binary files a/Lib/ensurepip/_bundled/pip-21.2.3-py3-none-any.whl and /dev/null differ diff --git a/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl new file mode 100644 index 0000000000000..46d3012c59b17 Binary files /dev/null and b/Lib/ensurepip/_bundled/pip-21.2.4-py3-none-any.whl differ diff --git a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl similarity index 80% rename from Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl rename to Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl index af8f8ba21003b..18c8c22958f1f 100644 Binary files a/Lib/ensurepip/_bundled/setuptools-57.4.0-py3-none-any.whl and b/Lib/ensurepip/_bundled/setuptools-58.1.0-py3-none-any.whl differ diff --git a/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst new file mode 100644 index 0000000000000..8dac4e6a25271 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-01-23-07-02.bpo-45343.ixmctD.rst @@ -0,0 +1 @@ +Update bundled pip to 21.2.4 and setuptools to 58.1.0 From webhook-mailer at python.org Tue Oct 5 22:03:05 2021 From: webhook-mailer at python.org (orsenthil) Date: Wed, 06 Oct 2021 02:03:05 -0000 Subject: [Python-checkins] bpo-40321: Support HTTP response status code 308 in urllib.request (#19588) Message-ID: https://github.com/python/cpython/commit/c379bc5ec9012cf66424ef3d80612cf13ec51006 commit: c379bc5ec9012cf66424ef3d80612cf13ec51006 branch: main author: Jochem Schulenklopper committer: orsenthil date: 2021-10-05T19:02:58-07:00 summary: bpo-40321: Support HTTP response status code 308 in urllib.request (#19588) * Support HTTP response status code 308 in urllib. HTTP response status code 308 is defined in https://tools.ietf.org/html/rfc7538 to be the permanent redirect variant of 307 (temporary redirect). * Update documentation to include http_error_308() * Add blurb for bpo-40321 fix Co-authored-by: Roland Crosby files: A Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst M Doc/library/urllib.request.rst M Lib/urllib/request.py diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 659a3632ac9be..099d74b2d5eab 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -879,6 +879,12 @@ HTTPRedirectHandler Objects response. +.. method:: HTTPRedirectHandler.http_error_308(req, fp, code, msg, hdrs) + + The same as :meth:`http_error_301`, but called for the 'permanent redirect' + response. + + .. _http-cookie-processor: HTTPCookieProcessor Objects diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index eca6cc350161f..3ba6d926aa8e7 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -11,8 +11,8 @@ Handlers needed to open the requested URL. For example, the HTTPHandler performs HTTP GET and POST requests and deals with non-error returns. The HTTPRedirectHandler automatically deals with -HTTP 301, 302, 303 and 307 redirect errors, and the HTTPDigestAuthHandler -deals with digest authentication. +HTTP 301, 302, 303, 307 and 308 redirect errors, and the +HTTPDigestAuthHandler deals with digest authentication. urlopen(url, data=None) -- Basic usage is the same as original urllib. pass the url and optionally data to post to an HTTP URL, and @@ -661,7 +661,7 @@ def redirect_request(self, req, fp, code, msg, headers, newurl): but another Handler might. """ m = req.get_method() - if (not (code in (301, 302, 303, 307) and m in ("GET", "HEAD") + if (not (code in (301, 302, 303, 307, 308) and m in ("GET", "HEAD") or code in (301, 302, 303) and m == "POST")): raise HTTPError(req.full_url, code, msg, headers, fp) @@ -748,7 +748,7 @@ def http_error_302(self, req, fp, code, msg, headers): return self.parent.open(new, timeout=req.timeout) - http_error_301 = http_error_303 = http_error_307 = http_error_302 + http_error_301 = http_error_303 = http_error_307 = http_error_308 = http_error_302 inf_msg = "The HTTP server returned a redirect error that would " \ "lead to an infinite loop.\n" \ @@ -2211,6 +2211,13 @@ def http_error_307(self, url, fp, errcode, errmsg, headers, data=None): else: return self.http_error_default(url, fp, errcode, errmsg, headers) + def http_error_308(self, url, fp, errcode, errmsg, headers, data=None): + """Error 308 -- relocated, but turn POST into error.""" + if data is None: + return self.http_error_301(url, fp, errcode, errmsg, headers, data) + else: + return self.http_error_default(url, fp, errcode, errmsg, headers) + def http_error_401(self, url, fp, errcode, errmsg, headers, data=None, retry=False): """Error 401 -- authentication required. diff --git a/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst b/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst new file mode 100644 index 0000000000000..1a7dba249c7db --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst @@ -0,0 +1,2 @@ +Adds support for HTTP 308 redirects to :mod:`urllib`. Patch by Jochem +Schulenklopper. From webhook-mailer at python.org Wed Oct 6 08:05:52 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 06 Oct 2021 12:05:52 -0000 Subject: [Python-checkins] Normalize jumps in compiler. All forward jumps to use JUMP_FORWARD. (GH-28755) Message-ID: https://github.com/python/cpython/commit/f6eafe18c004c55082de40d20cad084ef9dd3db7 commit: f6eafe18c004c55082de40d20cad084ef9dd3db7 branch: main author: Mark Shannon committer: markshannon date: 2021-10-06T13:05:45+01:00 summary: Normalize jumps in compiler. All forward jumps to use JUMP_FORWARD. (GH-28755) files: M Lib/test/test_dis.py M Python/ceval.c M Python/compile.c diff --git a/Lib/test/test_dis.py b/Lib/test/test_dis.py index 0899db66d8241..4755e3f3a1724 100644 --- a/Lib/test/test_dis.py +++ b/Lib/test/test_dis.py @@ -1046,7 +1046,7 @@ def _prepare_test_cases(): Instruction(opname='COMPARE_OP', opcode=107, arg=4, argval='>', argrepr='>', offset=34, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=21, argval=42, argrepr='to 42', offset=36, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='POP_TOP', opcode=1, arg=None, argval=None, argrepr='', offset=38, starts_line=8, is_jump_target=False, positions=None), - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=26, argval=52, argrepr='to 52', offset=40, starts_line=None, is_jump_target=False, positions=None), + Instruction(opname='JUMP_FORWARD', opcode=110, arg=5, argval=52, argrepr='to 52', offset=40, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=4, argval=8, argrepr='to 8', offset=42, starts_line=7, is_jump_target=True, positions=None), Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=44, starts_line=10, is_jump_target=True, positions=None), Instruction(opname='LOAD_CONST', opcode=100, arg=4, argval='I can haz else clause?', argrepr="'I can haz else clause?'", offset=46, starts_line=None, is_jump_target=False, positions=None), @@ -1071,7 +1071,7 @@ def _prepare_test_cases(): Instruction(opname='LOAD_CONST', opcode=100, arg=2, argval=4, argrepr='4', offset=84, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='COMPARE_OP', opcode=107, arg=0, argval='<', argrepr='<', offset=86, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='POP_JUMP_IF_FALSE', opcode=114, arg=46, argval=92, argrepr='to 92', offset=88, starts_line=None, is_jump_target=False, positions=None), - Instruction(opname='JUMP_ABSOLUTE', opcode=113, arg=52, argval=104, argrepr='to 104', offset=90, starts_line=17, is_jump_target=False, positions=None), + Instruction(opname='JUMP_FORWARD', opcode=110, arg=6, argval=104, argrepr='to 104', offset=90, starts_line=17, is_jump_target=False, positions=None), Instruction(opname='LOAD_FAST', opcode=124, arg=0, argval='i', argrepr='i', offset=92, starts_line=11, is_jump_target=True, positions=None), Instruction(opname='POP_JUMP_IF_TRUE', opcode=115, arg=28, argval=56, argrepr='to 56', offset=94, starts_line=None, is_jump_target=False, positions=None), Instruction(opname='LOAD_GLOBAL', opcode=116, arg=1, argval='print', argrepr='print', offset=96, starts_line=19, is_jump_target=True, positions=None), diff --git a/Python/ceval.c b/Python/ceval.c index 2dbc291897c03..a3a173dfb7013 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4051,19 +4051,18 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(JUMP_ABSOLUTE) { PREDICTED(JUMP_ABSOLUTE); - if (oparg < INSTR_OFFSET()) { - /* Increment the warmup counter and quicken if warm enough - * _Py_Quicken is idempotent so we don't worry about overflow */ - if (!PyCodeObject_IsWarmedUp(co)) { - PyCodeObject_IncrementWarmup(co); - if (PyCodeObject_IsWarmedUp(co)) { - if (_Py_Quicken(co)) { - goto error; - } - int nexti = INSTR_OFFSET(); - first_instr = co->co_firstinstr; - next_instr = first_instr + nexti; + assert(oparg < INSTR_OFFSET()); + /* Increment the warmup counter and quicken if warm enough + * _Py_Quicken is idempotent so we don't worry about overflow */ + if (!PyCodeObject_IsWarmedUp(co)) { + PyCodeObject_IncrementWarmup(co); + if (PyCodeObject_IsWarmedUp(co)) { + if (_Py_Quicken(co)) { + goto error; } + int nexti = INSTR_OFFSET(); + first_instr = co->co_firstinstr; + next_instr = first_instr + nexti; } } JUMPTO(oparg); @@ -4072,6 +4071,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } TARGET(JUMP_ABSOLUTE_QUICK) { + assert(oparg < INSTR_OFFSET()); JUMPTO(oparg); CHECK_EVAL_BREAKER(); DISPATCH(); diff --git a/Python/compile.c b/Python/compile.c index 694da29b771d0..2d82d6a1e5a91 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -7220,6 +7220,26 @@ assemble_emit(struct assembler *a, struct instr *i) return 1; } +static void +normalize_jumps(struct assembler *a) +{ + for (basicblock *b = a->a_entry; b != NULL; b = b->b_next) { + b->b_visited = 0; + } + for (basicblock *b = a->a_entry; b != NULL; b = b->b_next) { + b->b_visited = 1; + if (b->b_iused == 0) { + continue; + } + struct instr *last = &b->b_instr[b->b_iused-1]; + if (last->i_opcode == JUMP_ABSOLUTE && + last->i_target->b_visited == 0 + ) { + last->i_opcode = JUMP_FORWARD; + } + } +} + static void assemble_jump_offsets(struct assembler *a, struct compiler *c) { @@ -7897,6 +7917,9 @@ assemble(struct compiler *c, int addNone) clean_basic_block(b); } + /* Order of basic blocks must have been determined by now */ + normalize_jumps(&a); + /* Can't modify the bytecode after computing jump offsets. */ assemble_jump_offsets(&a, c); From webhook-mailer at python.org Wed Oct 6 08:20:02 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 06 Oct 2021 12:20:02 -0000 Subject: [Python-checkins] bpo-40116: Add insertion order bit-vector to dict values to allow dicts to share keys more freely. (GH-28520) Message-ID: https://github.com/python/cpython/commit/a7252f88d3fa33036bdd6036b8c97bc785ed6f17 commit: a7252f88d3fa33036bdd6036b8c97bc785ed6f17 branch: main author: Mark Shannon committer: markshannon date: 2021-10-06T13:19:53+01:00 summary: bpo-40116: Add insertion order bit-vector to dict values to allow dicts to share keys more freely. (GH-28520) files: A Misc/NEWS.d/next/Core and Builtins/2021-09-23-14-00-05.bpo-40116.KaoeFs.rst M Include/cpython/dictobject.h M Include/internal/pycore_dict.h M Lib/test/test_dict.py M Modules/_testcapimodule.c M Objects/dictobject.c M Python/ceval.c M Tools/gdb/libpython.py diff --git a/Include/cpython/dictobject.h b/Include/cpython/dictobject.h index 7c63374c566c7..ba118788f7b1b 100644 --- a/Include/cpython/dictobject.h +++ b/Include/cpython/dictobject.h @@ -3,6 +3,7 @@ #endif typedef struct _dictkeysobject PyDictKeysObject; +typedef struct _dictvalues PyDictValues; /* The ma_values pointer is NULL for a combined table * or points to an array of PyObject* for a split table @@ -24,7 +25,7 @@ typedef struct { If ma_values is not NULL, the table is splitted: keys are stored in ma_keys and values are stored in ma_values */ - PyObject **ma_values; + PyDictValues *ma_values; } PyDictObject; PyAPI_FUNC(PyObject *) _PyDict_GetItem_KnownHash(PyObject *mp, PyObject *key, diff --git a/Include/internal/pycore_dict.h b/Include/internal/pycore_dict.h index 2becc30beb4d8..d37ef71bbcad3 100644 --- a/Include/internal/pycore_dict.h +++ b/Include/internal/pycore_dict.h @@ -71,6 +71,14 @@ struct _dictkeysobject { see the DK_ENTRIES() macro */ }; +/* This must be no more than 16, for the order vector to fit in 64 bits */ +#define SHARED_KEYS_MAX_SIZE 16 + +struct _dictvalues { + uint64_t mv_order; + PyObject *values[1]; +}; + #define DK_LOG_SIZE(dk) ((dk)->dk_log2_size) #if SIZEOF_VOID_P > 4 #define DK_SIZE(dk) (((int64_t)1)<tp_name); - return NULL; - } - - return PyBool_FromLong(_PyDict_HasSplitTable((PyDictObject*)arg)); -} - /* Issue #4701: Check that PyObject_Hash implicitly calls * PyType_Ready if it hasn't already been called */ @@ -5721,7 +5708,6 @@ static PyMethodDef TestMethods[] = { {"test_list_api", test_list_api, METH_NOARGS}, {"test_dict_iteration", test_dict_iteration, METH_NOARGS}, {"dict_getitem_knownhash", dict_getitem_knownhash, METH_VARARGS}, - {"dict_hassplittable", dict_hassplittable, METH_O}, {"test_lazy_hash_inheritance", test_lazy_hash_inheritance,METH_NOARGS}, {"test_long_api", test_long_api, METH_NOARGS}, {"test_xincref_doesnt_leak",test_xincref_doesnt_leak, METH_NOARGS}, diff --git a/Objects/dictobject.c b/Objects/dictobject.c index ae0098be5b547..824cba949d7a8 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -60,7 +60,6 @@ The DictObject can be in one of two forms. ma_values != NULL, dk_refcnt >= 1 Values are stored in the ma_values array. Only string (unicode) keys are allowed. - All dicts sharing same key must have same insertion order. There are four kinds of slots in the table (slot is index, and DK_ENTRIES(keys)[index] if index >= 0): @@ -96,9 +95,10 @@ dk_nentries to achieve amortized O(1). Since there are DKIX_DUMMY remains in dk_indices, we can't increment dk_usable even though dk_nentries is decremented. -In split table, inserting into pending entry is allowed only for dk_entries[ix] -where ix == mp->ma_used. Inserting into other index and deleting item cause -converting the dict to the combined table. +To preserve the order in a split table, a bit vector is used to record the +insertion order. When a key is inserted the bit vector is shifted up by 4 bits +and the index of the key is stored in the low 4 bits. +As a consequence of this, split keys have a maximum size of 16. */ /* PyDict_MINSIZE is the starting size for any new dict. @@ -445,7 +445,9 @@ static PyDictKeysObject empty_keys_struct = { DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY, DKIX_EMPTY}, /* dk_indices */ }; -static PyObject *empty_values[1] = { NULL }; + +static PyDictValues empty_values_struct = { 0, { NULL }}; +#define empty_values (&empty_values_struct) #define Py_EMPTY_KEYS &empty_keys_struct @@ -458,6 +460,13 @@ static PyObject *empty_values[1] = { NULL }; # define ASSERT_CONSISTENT(op) assert(_PyDict_CheckConsistency((PyObject *)(op), 0)) #endif +static inline int +get_index_from_order(PyDictObject *mp, Py_ssize_t i) +{ + assert(mp->ma_used <= 16); + int shift = (int)(mp->ma_used-1-i)*4; + return (int)(mp->ma_values->mv_order >> shift) & 15; +} int _PyDict_CheckConsistency(PyObject *op, int check_content) @@ -482,17 +491,19 @@ _PyDict_CheckConsistency(PyObject *op, int check_content) /* combined table */ CHECK(keys->dk_refcnt == 1); } + else { + CHECK(mp->ma_used <= SHARED_KEYS_MAX_SIZE); + } if (check_content) { PyDictKeyEntry *entries = DK_ENTRIES(keys); - Py_ssize_t i; - for (i=0; i < DK_SIZE(keys); i++) { + for (Py_ssize_t i=0; i < DK_SIZE(keys); i++) { Py_ssize_t ix = dictkeys_get_index(keys, i); CHECK(DKIX_DUMMY <= ix && ix <= usable); } - for (i=0; i < usable; i++) { + for (Py_ssize_t i=0; i < usable; i++) { PyDictKeyEntry *entry = &entries[i]; PyObject *key = entry->me_key; @@ -517,9 +528,14 @@ _PyDict_CheckConsistency(PyObject *op, int check_content) } if (splitted) { + CHECK(mp->ma_used <= SHARED_KEYS_MAX_SIZE); /* splitted table */ - for (i=0; i < mp->ma_used; i++) { - CHECK(mp->ma_values[i] != NULL); + int duplicate_check = 0; + for (Py_ssize_t i=0; i < mp->ma_used; i++) { + int index = get_index_from_order(mp, i); + CHECK((duplicate_check & (1<ma_values->values[index] != NULL); } } } @@ -576,9 +592,9 @@ new_keys_object(uint8_t log2_size) #endif dk->dk_refcnt = 1; dk->dk_log2_size = log2_size; - dk->dk_usable = usable; dk->dk_kind = DICT_KEYS_UNICODE; dk->dk_nentries = 0; + dk->dk_usable = usable; dk->dk_version = 0; memset(&dk->dk_indices[0], 0xff, es<mv_order = 0; for (i = 0; i < size; i++) { - values[i] = NULL; + values->values[i] = NULL; } return new_dict(keys, values); } @@ -829,7 +852,7 @@ _Py_dict_lookup(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject **valu *value_addr = NULL; } else if (kind == DICT_KEYS_SPLIT) { - *value_addr = mp->ma_values[ix]; + *value_addr = mp->ma_values->values[ix]; } else { *value_addr = DK_ENTRIES(dk)[ix].me_value; @@ -879,7 +902,7 @@ _Py_dict_lookup(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject **valu Py_UNREACHABLE(); found: if (dk->dk_kind == DICT_KEYS_SPLIT) { - *value_addr = mp->ma_values[ix]; + *value_addr = mp->ma_values->values[ix]; } else { *value_addr = ep0[ix].me_value; @@ -928,7 +951,7 @@ _PyDict_MaybeUntrack(PyObject *op) numentries = mp->ma_keys->dk_nentries; if (_PyDict_HasSplitTable(mp)) { for (i = 0; i < numentries; i++) { - if ((value = mp->ma_values[i]) == NULL) + if ((value = mp->ma_values->values[i]) == NULL) continue; if (_PyObject_GC_MAY_BE_TRACKED(value)) { assert(!_PyObject_GC_MAY_BE_TRACKED(ep0[i].me_key)); @@ -998,17 +1021,6 @@ insertdict(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject *value) MAINTAIN_TRACKING(mp, key, value); - /* When insertion order is different from shared key, we can't share - * the key anymore. Convert this instance to combine table. - */ - if (_PyDict_HasSplitTable(mp) && - ((ix >= 0 && old_value == NULL && mp->ma_used != ix) || - (ix == DKIX_EMPTY && mp->ma_used != mp->ma_keys->dk_nentries))) { - if (insertion_resize(mp) < 0) - goto Fail; - ix = DKIX_EMPTY; - } - if (ix == DKIX_EMPTY) { /* Insert into new slot. */ mp->ma_keys->dk_version = 0; @@ -1027,8 +1039,12 @@ insertdict(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject *value) ep->me_key = key; ep->me_hash = hash; if (mp->ma_values) { - assert (mp->ma_values[mp->ma_keys->dk_nentries] == NULL); - mp->ma_values[mp->ma_keys->dk_nentries] = value; + Py_ssize_t index = mp->ma_keys->dk_nentries; + assert(index < SHARED_KEYS_MAX_SIZE); + assert((mp->ma_values->mv_order >> 60) == 0); + mp->ma_values->mv_order = (mp->ma_values->mv_order)<<4 | index; + assert (mp->ma_values->values[index] == NULL); + mp->ma_values->values[index] = value; } else { ep->me_value = value; @@ -1044,10 +1060,9 @@ insertdict(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject *value) if (old_value != value) { if (_PyDict_HasSplitTable(mp)) { - mp->ma_values[ix] = value; + mp->ma_values->values[ix] = value; if (old_value == NULL) { - /* pending state */ - assert(ix == mp->ma_used); + mp->ma_values->mv_order = (mp->ma_values->mv_order << 4) | ix; mp->ma_used++; } } @@ -1136,7 +1151,7 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) { Py_ssize_t numentries; PyDictKeysObject *oldkeys; - PyObject **oldvalues; + PyDictValues *oldvalues; PyDictKeyEntry *oldentries, *newentries; if (log2_newsize >= SIZEOF_SIZE_T*8) { @@ -1173,13 +1188,14 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) * Note that values of split table is always dense. */ for (Py_ssize_t i = 0; i < numentries; i++) { - assert(oldvalues[i] != NULL); - PyDictKeyEntry *ep = &oldentries[i]; + int index = oldvalues->mv_order >> ((numentries-1-i)*4) & 15; + assert(oldvalues->values[index] != NULL); + PyDictKeyEntry *ep = &oldentries[index]; PyObject *key = ep->me_key; Py_INCREF(key); newentries[i].me_key = key; newentries[i].me_hash = ep->me_hash; - newentries[i].me_value = oldvalues[i]; + newentries[i].me_value = oldvalues->values[index]; } dictkeys_decref(oldkeys); @@ -1238,9 +1254,12 @@ make_keys_shared(PyObject *op) if (!PyDict_CheckExact(op)) return NULL; + if (mp->ma_used > SHARED_KEYS_MAX_SIZE) { + return NULL; + } if (!_PyDict_HasSplitTable(mp)) { PyDictKeyEntry *ep0; - PyObject **values; + PyDictValues *values; assert(mp->ma_keys->dk_refcnt == 1); if (mp->ma_keys->dk_kind == DICT_KEYS_GENERAL) { return NULL; @@ -1260,14 +1279,29 @@ make_keys_shared(PyObject *op) "Not enough memory to allocate new values array"); return NULL; } - for (i = 0; i < size; i++) { - values[i] = ep0[i].me_value; + uint64_t order = 0; + for (i = 0; i < mp->ma_used; i++) { + order <<= 4; + order |= i; + assert(ep0[i].me_value != NULL); + values->values[i] = ep0[i].me_value; + ep0[i].me_value = NULL; + } + values->mv_order = order; + for (; i < size; i++) { + assert(ep0[i].me_value == NULL); + values->values[i] = NULL; ep0[i].me_value = NULL; } + if (mp->ma_keys->dk_nentries + mp->ma_keys->dk_usable > SHARED_KEYS_MAX_SIZE) { + assert(mp->ma_keys->dk_nentries <= SHARED_KEYS_MAX_SIZE); + mp->ma_keys->dk_usable = SHARED_KEYS_MAX_SIZE - mp->ma_keys->dk_nentries; + } mp->ma_keys->dk_kind = DICT_KEYS_SPLIT; mp->ma_values = values; } dictkeys_incref(mp->ma_keys); + ASSERT_CONSISTENT(mp); return mp->ma_keys; } @@ -1366,7 +1400,7 @@ _PyDict_GetItemHint(PyDictObject *mp, PyObject *key, if (ep->me_key == key) { if (mp->ma_keys->dk_kind == DICT_KEYS_SPLIT) { assert(mp->ma_values != NULL); - res = mp->ma_values[(size_t)hint]; + res = mp->ma_values->values[(size_t)hint]; } else { res = ep->me_value; @@ -1569,11 +1603,30 @@ delitem_common(PyDictObject *mp, Py_hash_t hash, Py_ssize_t ix, mp->ma_keys->dk_version = 0; mp->ma_version_tag = DICT_NEXT_VERSION(); ep = &DK_ENTRIES(mp->ma_keys)[ix]; - dictkeys_set_index(mp->ma_keys, hashpos, DKIX_DUMMY); - old_key = ep->me_key; - ep->me_key = NULL; - ep->me_value = NULL; - Py_DECREF(old_key); + if (mp->ma_values) { + assert(old_value == mp->ma_values->values[ix]); + mp->ma_values->values[ix] = NULL; + assert(ix < SHARED_KEYS_MAX_SIZE); + /* Update order */ + for (int i = 0;; i+= 4) { + assert (i < 64); + if (((mp->ma_values->mv_order >> i) & 15) == (uint64_t)ix) { + /* Remove 4 bits at ith position */ + uint64_t order = mp->ma_values->mv_order; + uint64_t high = ((order>>i)>>4)<ma_values->mv_order = high | low; + break; + } + } + } + else { + dictkeys_set_index(mp->ma_keys, hashpos, DKIX_DUMMY); + old_key = ep->me_key; + ep->me_key = NULL; + ep->me_value = NULL; + Py_DECREF(old_key); + } Py_DECREF(old_value); ASSERT_CONSISTENT(mp); @@ -1617,15 +1670,6 @@ _PyDict_DelItem_KnownHash(PyObject *op, PyObject *key, Py_hash_t hash) return -1; } - // Split table doesn't allow deletion. Combine it. - if (_PyDict_HasSplitTable(mp)) { - if (dictresize(mp, DK_LOG_SIZE(mp->ma_keys))) { - return -1; - } - ix = _Py_dict_lookup(mp, key, hash, &old_value); - assert(ix >= 0); - } - return delitem_common(mp, hash, ix, old_value); } @@ -1660,15 +1704,6 @@ _PyDict_DelItemIf(PyObject *op, PyObject *key, return -1; } - // Split table doesn't allow deletion. Combine it. - if (_PyDict_HasSplitTable(mp)) { - if (dictresize(mp, DK_LOG_SIZE(mp->ma_keys))) { - return -1; - } - ix = _Py_dict_lookup(mp, key, hash, &old_value); - assert(ix >= 0); - } - res = predicate(old_value); if (res == -1) return -1; @@ -1688,7 +1723,7 @@ PyDict_Clear(PyObject *op) { PyDictObject *mp; PyDictKeysObject *oldkeys; - PyObject **oldvalues; + PyDictValues *oldvalues; Py_ssize_t i, n; if (!PyDict_Check(op)) @@ -1708,7 +1743,7 @@ PyDict_Clear(PyObject *op) if (oldvalues != NULL) { n = oldkeys->dk_nentries; for (i = 0; i < n; i++) - Py_CLEAR(oldvalues[i]); + Py_CLEAR(oldvalues->values[i]); free_values(oldvalues); dictkeys_decref(oldkeys); } @@ -1738,11 +1773,12 @@ _PyDict_Next(PyObject *op, Py_ssize_t *ppos, PyObject **pkey, mp = (PyDictObject *)op; i = *ppos; if (mp->ma_values) { + assert(mp->ma_used <= SHARED_KEYS_MAX_SIZE); if (i < 0 || i >= mp->ma_used) return 0; - /* values of split table is always dense */ - entry_ptr = &DK_ENTRIES(mp->ma_keys)[i]; - value = mp->ma_values[i]; + int index = get_index_from_order(mp, i); + entry_ptr = &DK_ENTRIES(mp->ma_keys)[index]; + value = mp->ma_values->values[index]; assert(value != NULL); } else { @@ -1796,9 +1832,8 @@ PyDict_Next(PyObject *op, Py_ssize_t *ppos, PyObject **pkey, PyObject **pvalue) PyObject * _PyDict_Pop_KnownHash(PyObject *dict, PyObject *key, Py_hash_t hash, PyObject *deflt) { - Py_ssize_t ix, hashpos; - PyObject *old_value, *old_key; - PyDictKeyEntry *ep; + Py_ssize_t ix; + PyObject *old_value; PyDictObject *mp; assert(PyDict_Check(dict)); @@ -1823,29 +1858,9 @@ _PyDict_Pop_KnownHash(PyObject *dict, PyObject *key, Py_hash_t hash, PyObject *d _PyErr_SetKeyError(key); return NULL; } - - // Split table doesn't allow deletion. Combine it. - if (_PyDict_HasSplitTable(mp)) { - if (dictresize(mp, DK_LOG_SIZE(mp->ma_keys))) { - return NULL; - } - ix = _Py_dict_lookup(mp, key, hash, &old_value); - assert(ix >= 0); - } - - hashpos = lookdict_index(mp->ma_keys, hash, ix); - assert(hashpos >= 0); assert(old_value != NULL); - mp->ma_used--; - mp->ma_version_tag = DICT_NEXT_VERSION(); - mp->ma_keys->dk_version = 0; - dictkeys_set_index(mp->ma_keys, hashpos, DKIX_DUMMY); - ep = &DK_ENTRIES(mp->ma_keys)[ix]; - mp->ma_keys->dk_version = 0; - old_key = ep->me_key; - ep->me_key = NULL; - ep->me_value = NULL; - Py_DECREF(old_key); + Py_INCREF(old_value); + delitem_common(mp, hash, ix, old_value); ASSERT_CONSISTENT(mp); return old_value; @@ -1966,7 +1981,7 @@ _PyDict_FromKeys(PyObject *cls, PyObject *iterable, PyObject *value) static void dict_dealloc(PyDictObject *mp) { - PyObject **values = mp->ma_values; + PyDictValues *values = mp->ma_values; PyDictKeysObject *keys = mp->ma_keys; Py_ssize_t i, n; @@ -1976,7 +1991,7 @@ dict_dealloc(PyDictObject *mp) if (values != NULL) { if (values != empty_values) { for (i = 0, n = mp->ma_keys->dk_nentries; i < n; i++) { - Py_XDECREF(values[i]); + Py_XDECREF(values->values[i]); } free_values(values); } @@ -2165,7 +2180,7 @@ dict_keys(PyDictObject *mp) } ep = DK_ENTRIES(mp->ma_keys); if (mp->ma_values) { - value_ptr = mp->ma_values; + value_ptr = mp->ma_values->values; offset = sizeof(PyObject *); } else { @@ -2208,7 +2223,7 @@ dict_values(PyDictObject *mp) } ep = DK_ENTRIES(mp->ma_keys); if (mp->ma_values) { - value_ptr = mp->ma_values; + value_ptr = mp->ma_values->values; offset = sizeof(PyObject *); } else { @@ -2265,7 +2280,7 @@ dict_items(PyDictObject *mp) /* Nothing we do below makes any function calls. */ ep = DK_ENTRIES(mp->ma_keys); if (mp->ma_values) { - value_ptr = mp->ma_values; + value_ptr = mp->ma_values->values; offset = sizeof(PyObject *); } else { @@ -2534,7 +2549,7 @@ dict_merge(PyObject *a, PyObject *b, int override) key = entry->me_key; hash = entry->me_hash; if (other->ma_values) - value = other->ma_values[i]; + value = other->ma_values->values[i]; else value = entry->me_value; @@ -2677,7 +2692,7 @@ PyDict_Copy(PyObject *o) if (_PyDict_HasSplitTable(mp)) { PyDictObject *split_copy; Py_ssize_t size = USABLE_FRACTION(DK_SIZE(mp->ma_keys)); - PyObject **newvalues; + PyDictValues *newvalues; newvalues = new_values(size); if (newvalues == NULL) return PyErr_NoMemory(); @@ -2686,15 +2701,16 @@ PyDict_Copy(PyObject *o) free_values(newvalues); return NULL; } + newvalues->mv_order = mp->ma_values->mv_order; split_copy->ma_values = newvalues; split_copy->ma_keys = mp->ma_keys; split_copy->ma_used = mp->ma_used; split_copy->ma_version_tag = DICT_NEXT_VERSION(); dictkeys_incref(mp->ma_keys); for (i = 0, n = size; i < n; i++) { - PyObject *value = mp->ma_values[i]; + PyObject *value = mp->ma_values->values[i]; Py_XINCREF(value); - split_copy->ma_values[i] = value; + split_copy->ma_values->values[i] = value; } if (_PyObject_GC_IS_TRACKED(mp)) _PyObject_GC_TRACK(split_copy); @@ -2806,7 +2822,7 @@ dict_equal(PyDictObject *a, PyDictObject *b) PyDictKeyEntry *ep = &DK_ENTRIES(a->ma_keys)[i]; PyObject *aval; if (a->ma_values) - aval = a->ma_values[i]; + aval = a->ma_values->values[i]; else aval = ep->me_value; if (aval != NULL) { @@ -2993,8 +3009,11 @@ PyDict_SetDefault(PyObject *d, PyObject *key, PyObject *defaultobj) ep->me_key = key; ep->me_hash = hash; if (_PyDict_HasSplitTable(mp)) { - assert(mp->ma_values[mp->ma_keys->dk_nentries] == NULL); - mp->ma_values[mp->ma_keys->dk_nentries] = value; + int index = (int)mp->ma_keys->dk_nentries; + assert(index < SHARED_KEYS_MAX_SIZE); + assert(mp->ma_values->values[index] == NULL); + mp->ma_values->values[index] = value; + mp->ma_values->mv_order = (mp->ma_values->mv_order << 4) | index; } else { ep->me_value = value; @@ -3011,7 +3030,8 @@ PyDict_SetDefault(PyObject *d, PyObject *key, PyObject *defaultobj) assert(ix == mp->ma_used); Py_INCREF(value); MAINTAIN_TRACKING(mp, key, value); - mp->ma_values[ix] = value; + mp->ma_values->values[ix] = value; + mp->ma_values->mv_order = (mp->ma_values->mv_order << 4) | ix; mp->ma_used++; mp->ma_version_tag = DICT_NEXT_VERSION(); } @@ -3159,7 +3179,7 @@ dict_traverse(PyObject *op, visitproc visit, void *arg) else { if (mp->ma_values != NULL) { for (i = 0; i < n; i++) { - Py_VISIT(mp->ma_values[i]); + Py_VISIT(mp->ma_values->values[i]); } } else { @@ -3677,8 +3697,9 @@ dictiter_iternextkey(dictiterobject *di) if (d->ma_values) { if (i >= d->ma_used) goto fail; - key = DK_ENTRIES(k)[i].me_key; - assert(d->ma_values[i] != NULL); + int index = get_index_from_order(d, i); + key = DK_ENTRIES(k)[index].me_key; + assert(d->ma_values->values[index] != NULL); } else { Py_ssize_t n = k->dk_nentries; @@ -3764,7 +3785,8 @@ dictiter_iternextvalue(dictiterobject *di) if (d->ma_values) { if (i >= d->ma_used) goto fail; - value = d->ma_values[i]; + int index = get_index_from_order(d, i); + value = d->ma_values->values[index]; assert(value != NULL); } else { @@ -3851,8 +3873,9 @@ dictiter_iternextitem(dictiterobject *di) if (d->ma_values) { if (i >= d->ma_used) goto fail; - key = DK_ENTRIES(d->ma_keys)[i].me_key; - value = d->ma_values[i]; + int index = get_index_from_order(d, i); + key = DK_ENTRIES(d->ma_keys)[index].me_key; + value = d->ma_values->values[index]; assert(value != NULL); } else { @@ -3968,8 +3991,9 @@ dictreviter_iternext(dictiterobject *di) goto fail; } if (d->ma_values) { - key = DK_ENTRIES(k)[i].me_key; - value = d->ma_values[i]; + int index = get_index_from_order(d, i); + key = DK_ENTRIES(k)[index].me_key; + value = d->ma_values->values[index]; assert (value != NULL); } else { @@ -4976,19 +5000,14 @@ _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, } if (value == NULL) { res = PyDict_DelItem(dict, key); - // Since key sharing dict doesn't allow deletion, PyDict_DelItem() - // always converts dict to combined form. - if ((cached = CACHED_KEYS(tp)) != NULL) { - CACHED_KEYS(tp) = NULL; - dictkeys_decref(cached); - } } else { int was_shared = (cached == ((PyDictObject *)dict)->ma_keys); res = PyDict_SetItem(dict, key, value); if (was_shared && (cached = CACHED_KEYS(tp)) != NULL && - cached != ((PyDictObject *)dict)->ma_keys) { + cached != ((PyDictObject *)dict)->ma_keys && + cached->dk_nentries <= SHARED_KEYS_MAX_SIZE) { /* PyDict_SetItem() may call dictresize and convert split table * into combined table. In such case, convert it to split * table again and update type's shared key only when this is @@ -5004,14 +5023,15 @@ _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, * a = C() */ if (cached->dk_refcnt == 1) { - CACHED_KEYS(tp) = make_keys_shared(dict); - } - else { - CACHED_KEYS(tp) = NULL; + PyDictKeysObject *new_cached = make_keys_shared(dict); + if (new_cached != NULL) { + CACHED_KEYS(tp) = new_cached; + dictkeys_decref(cached); + } + else if (PyErr_Occurred()) { + return -1; + } } - dictkeys_decref(cached); - if (CACHED_KEYS(tp) == NULL && PyErr_Occurred()) - return -1; } } } else { @@ -5028,6 +5048,7 @@ _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, res = PyDict_SetItem(dict, key, value); } } + ASSERT_CONSISTENT(dict); return res; } diff --git a/Python/ceval.c b/Python/ceval.c index a3a173dfb7013..e39ec67614bf5 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3616,7 +3616,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(dict == NULL, LOAD_ATTR); assert(PyDict_CheckExact((PyObject *)dict)); DEOPT_IF(dict->ma_keys->dk_version != cache1->dk_version_or_hint, LOAD_ATTR); - res = dict->ma_values[cache0->index]; + res = dict->ma_values->values[cache0->index]; DEOPT_IF(res == NULL, LOAD_ATTR); STAT_INC(LOAD_ATTR, hit); record_cache_hit(cache0); @@ -3722,15 +3722,16 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(dict == NULL, STORE_ATTR); assert(PyDict_CheckExact((PyObject *)dict)); DEOPT_IF(dict->ma_keys->dk_version != cache1->dk_version_or_hint, STORE_ATTR); - /* Need to maintain ordering of dicts */ - DEOPT_IF(cache0->index > 0 && dict->ma_values[cache0->index-1] == NULL, STORE_ATTR); STAT_INC(STORE_ATTR, hit); record_cache_hit(cache0); + int index = cache0->index; STACK_SHRINK(1); PyObject *value = POP(); - PyObject *old_value = dict->ma_values[cache0->index]; - dict->ma_values[cache0->index] = value; + PyObject *old_value = dict->ma_values->values[index]; + dict->ma_values->values[index] = value; if (old_value == NULL) { + assert(index < 16); + dict->ma_values->mv_order = (dict->ma_values->mv_order << 4) | index; dict->ma_used++; } else { diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index c11b23e74b9be..62eb1976b715f 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -686,10 +686,13 @@ def iteritems(self): ''' keys = self.field('ma_keys') values = self.field('ma_values') + has_values = long(values) + if has_values: + values = values['values'] entries, nentries = self._get_entries(keys) for i in safe_range(nentries): ep = entries[i] - if long(values): + if has_values: pyop_value = PyObjectPtr.from_pyobject_ptr(values[i]) else: pyop_value = PyObjectPtr.from_pyobject_ptr(ep['me_value']) From webhook-mailer at python.org Wed Oct 6 09:04:56 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 13:04:56 -0000 Subject: [Python-checkins] [doc] Update references to NumPy (GH-22458) (GH-28749) Message-ID: https://github.com/python/cpython/commit/d747f5e805fa1c33768d9c22605e6324a35b3709 commit: d747f5e805fa1c33768d9c22605e6324a35b3709 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T15:04:48+02:00 summary: [doc] Update references to NumPy (GH-22458) (GH-28749) Numeric(al) Python to NumPy. It seems the old name hasn't been used for some time. (cherry picked from commit c8bb24166e367d449158015cb9b1093f03c7175d) Co-authored-by: Andre Delfino files: M Doc/faq/programming.rst M Doc/library/array.rst M Doc/library/functions.rst M Doc/tutorial/floatingpoint.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 04d6592aeccdb..4e04b10b0dae6 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -1184,7 +1184,7 @@ difference is that a Python list can contain objects of many different types. The ``array`` module also provides methods for creating arrays of fixed types with compact representations, but they are slower to index than lists. Also -note that the Numeric extensions and others define array-like structures with +note that NumPy and other third party packages define array-like structures with various characteristics as well. To get Lisp-style linked lists, you can emulate cons cells using tuples:: diff --git a/Doc/library/array.rst b/Doc/library/array.rst index f2f7894e1bf0f..f892d0983b6b3 100644 --- a/Doc/library/array.rst +++ b/Doc/library/array.rst @@ -256,7 +256,6 @@ Examples:: Packing and unpacking of External Data Representation (XDR) data as used in some remote procedure call systems. - `The Numerical Python Documentation `_ - The Numeric Python extension (NumPy) defines another array type; see - http://www.numpy.org/ for further information about Numerical Python. + `NumPy `_ + The NumPy package defines another array type. diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index 4f967825fcbbb..b17ca69760dbc 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1513,14 +1513,12 @@ are always available. They are listed here in alphabetical order. .. class:: slice(stop) slice(start, stop[, step]) - .. index:: single: Numerical Python - Return a :term:`slice` object representing the set of indices specified by ``range(start, stop, step)``. The *start* and *step* arguments default to ``None``. Slice objects have read-only data attributes :attr:`~slice.start`, :attr:`~slice.stop` and :attr:`~slice.step` which merely return the argument values (or their default). They have no other explicit functionality; - however they are used by Numerical Python and other third party extensions. + however they are used by NumPy and other third party packages. Slice objects are also generated when extended indexing syntax is used. For example: ``a[start:stop:step]`` or ``a[start:stop, i]``. See :func:`itertools.islice` for an alternate version that returns an iterator. diff --git a/Doc/tutorial/floatingpoint.rst b/Doc/tutorial/floatingpoint.rst index 0c0eb526fa9ed..b98de6e56a003 100644 --- a/Doc/tutorial/floatingpoint.rst +++ b/Doc/tutorial/floatingpoint.rst @@ -158,7 +158,7 @@ which implements arithmetic based on rational numbers (so the numbers like 1/3 can be represented exactly). If you are a heavy user of floating point operations you should take a look -at the Numerical Python package and many other packages for mathematical and +at the NumPy package and many other packages for mathematical and statistical operations supplied by the SciPy project. See . Python provides tools that may help on those rare occasions when you really From webhook-mailer at python.org Wed Oct 6 09:57:04 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 13:57:04 -0000 Subject: [Python-checkins] [3.10] [doc] Fix typos found using codespell (GH-28744) (GH-28758) Message-ID: https://github.com/python/cpython/commit/d15f47d1797292be7fe7f846f389bcd023a710d5 commit: d15f47d1797292be7fe7f846f389bcd023a710d5 branch: 3.10 author: Christian Clauss committer: ambv date: 2021-10-06T15:56:57+02:00 summary: [3.10] [doc] Fix typos found using codespell (GH-28744) (GH-28758) files: M Doc/c-api/call.rst M Doc/c-api/init_config.rst M Doc/faq/design.rst M Doc/library/ast.rst M Doc/library/base64.rst M Doc/library/fileinput.rst M Doc/library/importlib.metadata.rst M Doc/library/test.rst M Doc/library/types.rst M Doc/tools/rstlint.py M Doc/using/configure.rst M Doc/whatsnew/3.10.rst diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 31dc9c8031fdb..739b5e97d1515 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -185,7 +185,7 @@ Object Calling API Various functions are available for calling a Python object. Each converts its arguments to a convention supported by the called object ? either *tp_call* or vectorcall. -In order to do as litle conversion as possible, pick one that best fits +In order to do as little conversion as possible, pick one that best fits the format of data you have available. The following table summarizes the available functions; diff --git a/Doc/c-api/init_config.rst b/Doc/c-api/init_config.rst index fe5b83aa8dc95..c037f19ce64f3 100644 --- a/Doc/c-api/init_config.rst +++ b/Doc/c-api/init_config.rst @@ -22,7 +22,7 @@ There are two kinds of configuration: * The :ref:`Isolated Configuration ` can be used to embed Python into an application. It isolates Python from the system. For example, environments variables are ignored, the LC_CTYPE locale is left unchanged and - no signal handler is registred. + no signal handler is registered. The :c:func:`Py_RunMain` function can be used to write a customized Python program. @@ -696,7 +696,7 @@ PyConfig * Otherwise, use the :term:`locale encoding`: ``nl_langinfo(CODESET)`` result. - At Python statup, the encoding name is normalized to the Python codec + At Python startup, the encoding name is normalized to the Python codec name. For example, ``"ANSI_X3.4-1968"`` is replaced with ``"ascii"``. See also the :c:member:`~PyConfig.filesystem_errors` member. diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index 720b1e496eb84..0437b59d55da6 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -714,7 +714,7 @@ Why don't generators support the with statement? For technical reasons, a generator used directly as a context manager would not work correctly. When, as is most common, a generator is used as an iterator run to completion, no closing is needed. When it is, wrap -it as "contextlib.closing(generator)" in the 'with' statment. +it as "contextlib.closing(generator)" in the 'with' statement. Why are colons required for the if/while/def/class statements? diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index d84c841fa4a08..e29b5e88d71d4 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1920,7 +1920,7 @@ and classes for traversing abstract syntax trees: If source contains a null character ('\0'), :exc:`ValueError` is raised. .. warning:: - Note that succesfully parsing souce code into an AST object doesn't + Note that successfully parsing source code into an AST object doesn't guarantee that the source code provided is valid Python code that can be executed as the compilation step can raise further :exc:`SyntaxError` exceptions. For instance, the source ``return 42`` generates a valid diff --git a/Doc/library/base64.rst b/Doc/library/base64.rst index f91547bd58403..35fb7b69fa492 100644 --- a/Doc/library/base64.rst +++ b/Doc/library/base64.rst @@ -152,7 +152,7 @@ The modern interface provides: This version does not allow the digit 0 (zero) to the letter O (oh) and digit 1 (one) to either the letter I (eye) or letter L (el) mappings, all these characters are included in the Extended Hex Alphabet and are not - interchangable. + interchangeable. .. versionadded:: 3.10 diff --git a/Doc/library/fileinput.rst b/Doc/library/fileinput.rst index b840393292412..19cf7c67754f6 100644 --- a/Doc/library/fileinput.rst +++ b/Doc/library/fileinput.rst @@ -50,7 +50,7 @@ You can control how files are opened by providing an opening hook via the *openhook* parameter to :func:`fileinput.input` or :class:`FileInput()`. The hook must be a function that takes two arguments, *filename* and *mode*, and returns an accordingly opened file-like object. If *encoding* and/or *errors* -are specified, they will be passed to the hook as aditional keyword arguments. +are specified, they will be passed to the hook as additional keyword arguments. This module provides a :func:`hook_compressed` to support compressed files. The following function is the primary interface of this module: diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index c43457a385067..58c582d712413 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -255,7 +255,7 @@ function:: Package distributions --------------------- -A convience method to resolve the distribution or +A convenience method to resolve the distribution or distributions (in the case of a namespace package) for top-level Python packages or modules:: diff --git a/Doc/library/test.rst b/Doc/library/test.rst index a6cc2be4d5522..a8dc35476fc9b 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -1249,7 +1249,7 @@ The :mod:`test.support.threading_helper` module provides support for threading t Context manager catching :class:`threading.Thread` exception using :func:`threading.excepthook`. - Attributes set when an exception is catched: + Attributes set when an exception is caught: * ``exc_type`` * ``exc_value`` @@ -1458,7 +1458,7 @@ The :mod:`test.support.os_helper` module provides support for os tests. .. function:: unlink(filename) Call :func:`os.unlink` on *filename*. On Windows platforms, this is - wrapped with a wait loop that checks for the existence fo the file. + wrapped with a wait loop that checks for the existence of the file. :mod:`test.support.import_helper` --- Utilities for import tests diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 88fe47a8d4a7f..2314b02c7449c 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -243,7 +243,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__loader__", None)`` if you explicitly need to use this attribute. @@ -268,7 +268,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__package__", None)`` if you explicitly need to use this attribute. diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index cbcb8eb801b13..3092a3b2d81b7 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -165,7 +165,7 @@ def hide_comments(lines): """Tool to remove comments from given lines. It yields empty lines in place of comments, so line numbers are - still meaningfull. + still meaningful. """ in_multiline_comment = False for line in lines: diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index 177f25f318bbf..14ffdd760f012 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -550,7 +550,7 @@ Built-in modules have no ``__file__`` attribute:: File "", line 1, in AttributeError: module 'sys' has no attribute '__file__' -Other C extensins are built as dynamic libraires, like the ``_asyncio`` module. +Other C extensins are built as dynamic libraries, like the ``_asyncio`` module. They are built with the ``Py_BUILD_CORE_MODULE`` macro defined. Example on Linux x86-64:: diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index c11fe41659c20..803ba87b047e7 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1579,7 +1579,7 @@ Deprecated * Currently Python accepts numeric literals immediately followed by keywords, for example ``0in x``, ``1or x``, ``0if 1else 2``. It allows confusing - and ambigious expressions like ``[0x1for x in y]`` (which can be + and ambiguous expressions like ``[0x1for x in y]`` (which can be interpreted as ``[0x1 for x in y]`` or ``[0x1f or x in y]``). Starting in this release, a deprecation warning is raised if the numeric literal is immediately followed by one of keywords :keyword:`and`, :keyword:`else`, @@ -1916,7 +1916,7 @@ Changes in the Python API if the *globals* dictionary has no ``"__builtins__"`` key, rather than using ``{"None": None}`` as builtins: same behavior as :func:`eval` and :func:`exec` functions. Defining a function with ``def function(...): ...`` - in Python is not affected, globals cannot be overriden with this syntax: it + in Python is not affected, globals cannot be overridden with this syntax: it also inherits the current builtins. (Contributed by Victor Stinner in :issue:`42990`.) From webhook-mailer at python.org Wed Oct 6 09:57:44 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 13:57:44 -0000 Subject: [Python-checkins] [3.9] [doc] Fix typos found using codespell (GH-28744) (GH-28759) Message-ID: https://github.com/python/cpython/commit/edef03aaa8993901e63e901448ec2be8b2801ca7 commit: edef03aaa8993901e63e901448ec2be8b2801ca7 branch: 3.9 author: Christian Clauss committer: ambv date: 2021-10-06T15:57:39+02:00 summary: [3.9] [doc] Fix typos found using codespell (GH-28744) (GH-28759) files: M Doc/c-api/call.rst M Doc/faq/design.rst M Doc/library/ast.rst M Doc/library/test.rst M Doc/library/types.rst diff --git a/Doc/c-api/call.rst b/Doc/c-api/call.rst index 31dc9c8031fdb..739b5e97d1515 100644 --- a/Doc/c-api/call.rst +++ b/Doc/c-api/call.rst @@ -185,7 +185,7 @@ Object Calling API Various functions are available for calling a Python object. Each converts its arguments to a convention supported by the called object ? either *tp_call* or vectorcall. -In order to do as litle conversion as possible, pick one that best fits +In order to do as little conversion as possible, pick one that best fits the format of data you have available. The following table summarizes the available functions; diff --git a/Doc/faq/design.rst b/Doc/faq/design.rst index d2b868ebb8290..d0aee4e607268 100644 --- a/Doc/faq/design.rst +++ b/Doc/faq/design.rst @@ -715,7 +715,7 @@ Why don't generators support the with statement? For technical reasons, a generator used directly as a context manager would not work correctly. When, as is most common, a generator is used as an iterator run to completion, no closing is needed. When it is, wrap -it as "contextlib.closing(generator)" in the 'with' statment. +it as "contextlib.closing(generator)" in the 'with' statement. Why are colons required for the if/while/def/class statements? diff --git a/Doc/library/ast.rst b/Doc/library/ast.rst index 39a08416bad72..149179d5bfd5d 100644 --- a/Doc/library/ast.rst +++ b/Doc/library/ast.rst @@ -1552,7 +1552,7 @@ and classes for traversing abstract syntax trees: If source contains a null character ('\0'), :exc:`ValueError` is raised. .. warning:: - Note that succesfully parsing souce code into an AST object doesn't + Note that successfully parsing source code into an AST object doesn't guarantee that the source code provided is valid Python code that can be executed as the compilation step can raise further :exc:`SyntaxError` exceptions. For instance, the source ``return 42`` generates a valid diff --git a/Doc/library/test.rst b/Doc/library/test.rst index c4abcfffa1f14..16f908c8e8708 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -457,7 +457,7 @@ The :mod:`test.support` module defines the following functions: .. function:: unlink(filename) Call :func:`os.unlink` on *filename*. On Windows platforms, this is - wrapped with a wait loop that checks for the existence fo the file. + wrapped with a wait loop that checks for the existence of the file. .. function:: rmdir(filename) @@ -1145,7 +1145,7 @@ The :mod:`test.support` module defines the following functions: Context manager catching :class:`threading.Thread` exception using :func:`threading.excepthook`. - Attributes set when an exception is catched: + Attributes set when an exception is caught: * ``exc_type`` * ``exc_value`` diff --git a/Doc/library/types.rst b/Doc/library/types.rst index 81a2b7b987970..b6df39258e66e 100644 --- a/Doc/library/types.rst +++ b/Doc/library/types.rst @@ -229,7 +229,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__loader__", None)`` if you explicitly need to use this attribute. @@ -254,7 +254,7 @@ Standard names are defined for the following types: .. note:: A future version of Python may stop setting this attribute by default. - To guard against this potential change, preferrably read from the + To guard against this potential change, preferably read from the :attr:`__spec__` attribute instead or use ``getattr(module, "__package__", None)`` if you explicitly need to use this attribute. From webhook-mailer at python.org Wed Oct 6 10:52:05 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 14:52:05 -0000 Subject: [Python-checkins] bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) Message-ID: https://github.com/python/cpython/commit/7af95a1e8097b2aab2cbe8de88727809e745b658 commit: 7af95a1e8097b2aab2cbe8de88727809e745b658 branch: main author: DonnaDia <37962843+DonnaDia at users.noreply.github.com> committer: ambv date: 2021-10-06T16:51:55+02:00 summary: bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) Co-authored-by: ?ukasz Langa files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 74e861480d2ff..c7f8bc8f17f43 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -65,11 +65,10 @@ output must only depend on its input. Some languages are very strict about purity and don't even have assignment statements such as ``a=3`` or ``c = a + b``, but it's difficult to avoid all -side effects. Printing to the screen or writing to a disk file are side -effects, for example. For example, in Python a call to the :func:`print` or -:func:`time.sleep` function both return no useful value; they're only called for -their side effects of sending some text to the screen or pausing execution for a -second. +side effects, such as printing to the screen or writing to a disk file. Another +example is a call to the :func:`print` or :func:`time.sleep` function, neither +of which returns a useful value. Both are called only for their side effects +of sending some text to the screen or pausing execution for a second. Python programs written in functional style usually won't go to the extreme of avoiding all I/O or all assignments; instead, they'll provide a From webhook-mailer at python.org Wed Oct 6 11:04:07 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 15:04:07 -0000 Subject: [Python-checkins] Remove test_nntplib from quicktest (GH-28754) Message-ID: https://github.com/python/cpython/commit/61892c04764e1f3a659bbd09e6373687a27d36e2 commit: 61892c04764e1f3a659bbd09e6373687a27d36e2 branch: main author: Inada Naoki committer: ambv date: 2021-10-06T17:03:58+02:00 summary: Remove test_nntplib from quicktest (GH-28754) files: M Makefile.pre.in diff --git a/Makefile.pre.in b/Makefile.pre.in index ce75af1b79c81..4d4076f388260 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1345,7 +1345,7 @@ QUICKTESTOPTS= $(TESTOPTS) -x test_subprocess test_io test_lib2to3 \ test_multibytecodec test_urllib2_localnet test_itertools \ test_multiprocessing_fork test_multiprocessing_spawn \ test_multiprocessing_forkserver \ - test_mailbox test_socket test_poll \ + test_mailbox test_nntplib test_socket test_poll \ test_select test_zipfile test_concurrent_futures quicktest: @DEF_MAKE_RULE@ platform $(TESTRUNNER) $(QUICKTESTOPTS) From webhook-mailer at python.org Wed Oct 6 11:15:52 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 15:15:52 -0000 Subject: [Python-checkins] bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) (GH-28762) Message-ID: https://github.com/python/cpython/commit/dcdeb96495fa105098544e2be7b74fa288589912 commit: dcdeb96495fa105098544e2be7b74fa288589912 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T17:15:42+02:00 summary: bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) (GH-28762) Co-authored-by: ?ukasz Langa (cherry picked from commit 7af95a1e8097b2aab2cbe8de88727809e745b658) Co-authored-by: DonnaDia <37962843+DonnaDia at users.noreply.github.com> files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 74e861480d2ff..c7f8bc8f17f43 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -65,11 +65,10 @@ output must only depend on its input. Some languages are very strict about purity and don't even have assignment statements such as ``a=3`` or ``c = a + b``, but it's difficult to avoid all -side effects. Printing to the screen or writing to a disk file are side -effects, for example. For example, in Python a call to the :func:`print` or -:func:`time.sleep` function both return no useful value; they're only called for -their side effects of sending some text to the screen or pausing execution for a -second. +side effects, such as printing to the screen or writing to a disk file. Another +example is a call to the :func:`print` or :func:`time.sleep` function, neither +of which returns a useful value. Both are called only for their side effects +of sending some text to the screen or pausing execution for a second. Python programs written in functional style usually won't go to the extreme of avoiding all I/O or all assignments; instead, they'll provide a From webhook-mailer at python.org Wed Oct 6 11:19:56 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 15:19:56 -0000 Subject: [Python-checkins] bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) (GH-28763) Message-ID: https://github.com/python/cpython/commit/496d1aa0b84466cc9b11f4f3b90cee93af1f393e commit: 496d1aa0b84466cc9b11f4f3b90cee93af1f393e branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T17:19:46+02:00 summary: bpo-34804: [doc] Rephrase section on side effects in functional.rst for clarity (GH-27989) (GH-28763) Co-authored-by: ?ukasz Langa (cherry picked from commit 7af95a1e8097b2aab2cbe8de88727809e745b658) Co-authored-by: DonnaDia <37962843+DonnaDia at users.noreply.github.com> files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index 74e861480d2ff..c7f8bc8f17f43 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -65,11 +65,10 @@ output must only depend on its input. Some languages are very strict about purity and don't even have assignment statements such as ``a=3`` or ``c = a + b``, but it's difficult to avoid all -side effects. Printing to the screen or writing to a disk file are side -effects, for example. For example, in Python a call to the :func:`print` or -:func:`time.sleep` function both return no useful value; they're only called for -their side effects of sending some text to the screen or pausing execution for a -second. +side effects, such as printing to the screen or writing to a disk file. Another +example is a call to the :func:`print` or :func:`time.sleep` function, neither +of which returns a useful value. Both are called only for their side effects +of sending some text to the screen or pausing execution for a second. Python programs written in functional style usually won't go to the extreme of avoiding all I/O or all assignments; instead, they'll provide a From webhook-mailer at python.org Wed Oct 6 11:28:25 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 15:28:25 -0000 Subject: [Python-checkins] bpo-40321: Add missing test, slightly expand documentation (GH-28760) Message-ID: https://github.com/python/cpython/commit/f528045f695f7483d955a1eae4c1df68b1b4cacd commit: f528045f695f7483d955a1eae4c1df68b1b4cacd branch: main author: ?ukasz Langa committer: ambv date: 2021-10-06T17:28:16+02:00 summary: bpo-40321: Add missing test, slightly expand documentation (GH-28760) files: M Doc/library/urllib.request.rst M Lib/test/test_urllib2.py M Lib/urllib/request.py M Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst diff --git a/Doc/library/urllib.request.rst b/Doc/library/urllib.request.rst index 099d74b2d5eab..88e93ba6b002e 100644 --- a/Doc/library/urllib.request.rst +++ b/Doc/library/urllib.request.rst @@ -876,13 +876,17 @@ HTTPRedirectHandler Objects .. method:: HTTPRedirectHandler.http_error_307(req, fp, code, msg, hdrs) The same as :meth:`http_error_301`, but called for the 'temporary redirect' - response. + response. It does not allow changing the request method from ``POST`` + to ``GET``. .. method:: HTTPRedirectHandler.http_error_308(req, fp, code, msg, hdrs) The same as :meth:`http_error_301`, but called for the 'permanent redirect' - response. + response. It does not allow changing the request method from ``POST`` + to ``GET``. + + .. versionadded:: 3.11 .. _http-cookie-processor: diff --git a/Lib/test/test_urllib2.py b/Lib/test/test_urllib2.py index 9db23e6ce04bd..a2b1340e0bf5b 100644 --- a/Lib/test/test_urllib2.py +++ b/Lib/test/test_urllib2.py @@ -1163,7 +1163,7 @@ def test_redirect(self): o = h.parent = MockOpener() # ordinary redirect behaviour - for code in 301, 302, 303, 307: + for code in 301, 302, 303, 307, 308: for data in None, "blah\nblah\n": method = getattr(h, "http_error_%s" % code) req = Request(from_url, data) @@ -1176,8 +1176,8 @@ def test_redirect(self): method(req, MockFile(), code, "Blah", MockHeaders({"location": to_url})) except urllib.error.HTTPError: - # 307 in response to POST requires user OK - self.assertEqual(code, 307) + # 307 and 308 in response to POST require user OK + self.assertIn(code, (307, 308)) self.assertIsNotNone(data) self.assertEqual(o.req.get_full_url(), to_url) try: diff --git a/Lib/urllib/request.py b/Lib/urllib/request.py index 3ba6d926aa8e7..fd6fc36aee04b 100644 --- a/Lib/urllib/request.py +++ b/Lib/urllib/request.py @@ -11,7 +11,7 @@ Handlers needed to open the requested URL. For example, the HTTPHandler performs HTTP GET and POST requests and deals with non-error returns. The HTTPRedirectHandler automatically deals with -HTTP 301, 302, 303, 307 and 308 redirect errors, and the +HTTP 301, 302, 303, 307, and 308 redirect errors, and the HTTPDigestAuthHandler deals with digest authentication. urlopen(url, data=None) -- Basic usage is the same as original diff --git a/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst b/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst index 1a7dba249c7db..fede2a0e5e35f 100644 --- a/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst +++ b/Misc/NEWS.d/next/Library/2021-07-22-21-25-56.bpo-40321.gBlFmw.rst @@ -1,2 +1,2 @@ -Adds support for HTTP 308 redirects to :mod:`urllib`. Patch by Jochem -Schulenklopper. +Adds support for HTTP 308 redirects to :mod:`urllib`. See :rfc:`7538` for +details. Patch by Jochem Schulenklopper. From webhook-mailer at python.org Wed Oct 6 13:40:18 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 17:40:18 -0000 Subject: [Python-checkins] [Misc] [Mac] Fix typos found using codespell (GH-28756) Message-ID: https://github.com/python/cpython/commit/470145f572b53fe73518cda1eeacc56fec78c1b2 commit: 470145f572b53fe73518cda1eeacc56fec78c1b2 branch: main author: Christian Clauss committer: ambv date: 2021-10-06T19:40:09+02:00 summary: [Misc] [Mac] Fix typos found using codespell (GH-28756) files: M Mac/BuildScript/build-installer.py M Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py M Mac/PythonLauncher/MyAppDelegate.m M Mac/README.rst M Misc/NEWS.d/3.10.0a1.rst M Misc/NEWS.d/3.10.0a2.rst M Misc/NEWS.d/3.10.0a3.rst M Misc/NEWS.d/3.10.0a4.rst M Misc/NEWS.d/3.10.0a5.rst M Misc/NEWS.d/3.10.0a6.rst M Misc/NEWS.d/3.10.0a7.rst M Misc/NEWS.d/3.10.0b1.rst M Misc/NEWS.d/3.11.0a1.rst M Misc/NEWS.d/3.9.0a1.rst M Misc/NEWS.d/3.9.0a3.rst M Misc/NEWS.d/3.9.0a5.rst M Misc/stable_abi.txt diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 529060d8a1303..a6d5c349c348b 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -454,7 +454,7 @@ def pkg_recipes(): source="/pydocs", readme="""\ This package installs the python documentation at a location - that is useable for pydoc and IDLE. + that is usable for pydoc and IDLE. """, postflight="scripts/postflight.documentation", required=False, @@ -1602,7 +1602,7 @@ def buildDMG(): # instead of 11. We should not run into that situation here.) # Also we should use "macos" instead of "macosx" going forward. # - # To maintain compability for legacy variants, the file name for + # To maintain compatibility for legacy variants, the file name for # builds on macOS 10.15 and earlier remains: # python-3.x.y-macosx10.z.{dmg->pkg} # e.g. python-3.9.4-macosx10.9.{dmg->pkg} diff --git a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py index 5994c18ff876b..ccc18d4aee438 100644 --- a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py +++ b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py @@ -35,7 +35,7 @@ # idlemain.py running under the symlinked python. # This is the magic step. # 4. During interpreter initialization, because PYTHONEXECUTABLE is defined, -# sys.executable may get set to an unuseful value. +# sys.executable may get set to an useless value. # # (Note that the IDLE script and the setting of PYTHONEXECUTABLE is # generated automatically by bundlebuilder in the Python 2.x build. diff --git a/Mac/PythonLauncher/MyAppDelegate.m b/Mac/PythonLauncher/MyAppDelegate.m index 25779a2540a37..9cc2aa0ad9098 100644 --- a/Mac/PythonLauncher/MyAppDelegate.m +++ b/Mac/PythonLauncher/MyAppDelegate.m @@ -22,7 +22,7 @@ - (void)applicationDidFinishLaunching:(NSNotification *)notification { // Test that the file mappings are correct [self testFileTypeBinding]; - // If we were opened because of a file drag or doubleclick + // If we were opened because of a file drag or double-click // we've set initial_action_done in shouldShowUI // Otherwise we open a preferences dialog. if (!initial_action_done) { diff --git a/Mac/README.rst b/Mac/README.rst index 7dd41a3ee4725..7476639d0ff54 100644 --- a/Mac/README.rst +++ b/Mac/README.rst @@ -296,7 +296,7 @@ How do I create a binary distribution? Download and unpack the source release from https://www.python.org/download/. Go to the directory ``Mac/BuildScript``. There you will find a script ``build-installer.py`` that does all the work. This will download and build -a number of 3rd-party libaries, configures and builds a framework Python, +a number of 3rd-party libraries, configures and builds a framework Python, installs it, creates the installer package files and then packs this in a DMG image. The script also builds an HTML copy of the current Python documentation set for this release for inclusion in the framework. The diff --git a/Misc/NEWS.d/3.10.0a1.rst b/Misc/NEWS.d/3.10.0a1.rst index 044bd20594cc3..1c1c2d54e8c20 100644 --- a/Misc/NEWS.d/3.10.0a1.rst +++ b/Misc/NEWS.d/3.10.0a1.rst @@ -178,7 +178,7 @@ Convert the :mod:`_sha256` extension module types to heap types. .. section: Core and Builtins Fix a possible stack overflow in the parser when parsing functions and -classes with a huge ammount of arguments. Patch by Pablo Galindo. +classes with a huge amount of arguments. Patch by Pablo Galindo. .. diff --git a/Misc/NEWS.d/3.10.0a2.rst b/Misc/NEWS.d/3.10.0a2.rst index 3cfef17160336..61a291914f933 100644 --- a/Misc/NEWS.d/3.10.0a2.rst +++ b/Misc/NEWS.d/3.10.0a2.rst @@ -258,7 +258,7 @@ The :func:`repr` of :mod:`typing` types containing :ref:`Generic Alias Types .. nonce: 6aDbty .. section: Library -``binhex.binhex()`` consisently writes macOS 9 line endings. +``binhex.binhex()`` consistently writes macOS 9 line endings. .. @@ -534,7 +534,7 @@ belong to. Patch by Batuhan Taskaya. .. nonce: LR4fnY .. section: Library -Handle exceptions caused by unparseable date headers when using email +Handle exceptions caused by unparsable date headers when using email "default" policy. Patch by Tim Bell, Georges Toth .. diff --git a/Misc/NEWS.d/3.10.0a3.rst b/Misc/NEWS.d/3.10.0a3.rst index 0b76367f94445..4f182e8e3f1f0 100644 --- a/Misc/NEWS.d/3.10.0a3.rst +++ b/Misc/NEWS.d/3.10.0a3.rst @@ -949,7 +949,7 @@ branches, are now handled by docsbuild-script. .. nonce: W_updK .. section: Tests -Reenable test_gdb on gdb 9.2 and newer: +Re-enable test_gdb on gdb 9.2 and newer: https://bugzilla.redhat.com/show_bug.cgi?id=1866884 bug is fixed in gdb 10.1. diff --git a/Misc/NEWS.d/3.10.0a4.rst b/Misc/NEWS.d/3.10.0a4.rst index ff16a7037277c..beac530fcb28f 100644 --- a/Misc/NEWS.d/3.10.0a4.rst +++ b/Misc/NEWS.d/3.10.0a4.rst @@ -767,7 +767,7 @@ results. Patch by Ammar Askar. .. nonce: -7-XGz .. section: Tests -Update test_nntplib to use offical group name of news.aioe.org for testing. +Update test_nntplib to use official group name of news.aioe.org for testing. Patch by Dong-hee Na. .. @@ -863,7 +863,7 @@ and Peixing Xin. Now all platforms use a value for the "EXT_SUFFIX" build variable derived from SOABI (for instance in freeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" -instead of ".so"). Previosuly only Linux, Mac and VxWorks were using a value +instead of ".so"). Previously only Linux, Mac and VxWorks were using a value for "EXT_SUFFIX" that included "SOABI". .. diff --git a/Misc/NEWS.d/3.10.0a5.rst b/Misc/NEWS.d/3.10.0a5.rst index 989edb0b47e97..1c7c7447cae06 100644 --- a/Misc/NEWS.d/3.10.0a5.rst +++ b/Misc/NEWS.d/3.10.0a5.rst @@ -549,7 +549,7 @@ Pass ``--timeout=$(TESTTIMEOUT)`` option to the default profile task .. section: Build Removed the grep -q and -E flags in the tzpath validation section of the -configure script to better accomodate users of some platforms (specifically +configure script to better accommodate users of some platforms (specifically Solaris 10). .. diff --git a/Misc/NEWS.d/3.10.0a6.rst b/Misc/NEWS.d/3.10.0a6.rst index 46d06add9115f..a4ee9ae098bd9 100644 --- a/Misc/NEWS.d/3.10.0a6.rst +++ b/Misc/NEWS.d/3.10.0a6.rst @@ -49,7 +49,7 @@ The :data:`types.FunctionType` constructor now inherits the current builtins if the *globals* dictionary has no ``"__builtins__"`` key, rather than using ``{"None": None}`` as builtins: same behavior as :func:`eval` and :func:`exec` functions. Defining a function with ``def function(...): ...`` -in Python is not affected, globals cannot be overriden with this syntax: it +in Python is not affected, globals cannot be overridden with this syntax: it also inherits the current builtins. Patch by Victor Stinner. .. @@ -355,7 +355,7 @@ in 4.0" to "3.12". See :pep:`623` for detail. .. nonce: LfTvL- .. section: Tests -Fix test_importlib to correctly skip Unicode file tests if the fileystem +Fix test_importlib to correctly skip Unicode file tests if the filesystem does not support them. .. diff --git a/Misc/NEWS.d/3.10.0a7.rst b/Misc/NEWS.d/3.10.0a7.rst index a20072b7ae11e..6c32e60dc8a46 100644 --- a/Misc/NEWS.d/3.10.0a7.rst +++ b/Misc/NEWS.d/3.10.0a7.rst @@ -195,7 +195,7 @@ decoded as Unicode characters outside the [U+0000; U+10ffff] range. .. nonce: lCzIg0 .. section: Core and Builtins -Fix a bug that was causing the parser to crash when emiting syntax errors +Fix a bug that was causing the parser to crash when emitting syntax errors when reading input from stdin. Patch by Pablo Galindo .. diff --git a/Misc/NEWS.d/3.10.0b1.rst b/Misc/NEWS.d/3.10.0b1.rst index 25c3943abe50e..e4391a1ee3870 100644 --- a/Misc/NEWS.d/3.10.0b1.rst +++ b/Misc/NEWS.d/3.10.0b1.rst @@ -276,7 +276,7 @@ cause any runtime effects with ``from __future__ import annotations``. .. nonce: 0Ik1AM .. section: Core and Builtins -:exc:`SyntaxError` exceptions raised by the intepreter will highlight the +:exc:`SyntaxError` exceptions raised by the interpreter will highlight the full error range of the expression that consistutes the syntax error itself, instead of just where the problem is detected. Patch by Pablo Galindo. @@ -547,7 +547,7 @@ enum or one of its members' value. .. nonce: ejjsyR .. section: Library -For backwards compatbility with previous minor versions of Python, if +For backwards compatibility with previous minor versions of Python, if :func:`typing.get_type_hints` receives no namespace dictionary arguments, :func:`typing.get_type_hints` will search through the global then local namespaces during evaluation of stringized type annotations (string forward @@ -720,9 +720,9 @@ now raise ``TypeError`` during substitution. .. nonce: xT9QjF .. section: Library -The :mod:`multiprocessing` ``Server`` class now explicitly catchs +The :mod:`multiprocessing` ``Server`` class now explicitly catches :exc:`SystemExit` and closes the client connection in this case. It happens -when the ``Server.serve_client()`` method reachs the end of file (EOF). +when the ``Server.serve_client()`` method reaches the end of file (EOF). .. @@ -1132,7 +1132,7 @@ preferred "user", "home", and "prefix" (default) scheme names. .. section: Library Improve :meth:`sqlite3.Connection.backup` error handling. The error message -for non-existant target database names is now ``unknown database `` instead of ``SQL logic error``. Patch by Erlend E. Aasland. .. diff --git a/Misc/NEWS.d/3.11.0a1.rst b/Misc/NEWS.d/3.11.0a1.rst index ba07ef95b4801..bdfdab078c090 100644 --- a/Misc/NEWS.d/3.11.0a1.rst +++ b/Misc/NEWS.d/3.11.0a1.rst @@ -162,7 +162,7 @@ called directly or via ``super()``. Patch provided by Yurii Karabas. .. section: Core and Builtins The deallocator function of the :exc:`BaseException` type now uses the -trashcan mecanism to prevent stack overflow. For example, when a +trashcan mechanism to prevent stack overflow. For example, when a :exc:`RecursionError` instance is raised, it can be linked to another RecursionError through the ``__context__`` attribute or the ``__traceback__`` attribute, and then a chain of exceptions is created. When @@ -1138,7 +1138,7 @@ blocks. Patch by Pablo Galindo. .. nonce: vYFPPC .. section: Core and Builtins -Constructors of subclasses of some buitin classes (e.g. :class:`tuple`, +Constructors of subclasses of some builtin classes (e.g. :class:`tuple`, :class:`list`, :class:`frozenset`) no longer accept arbitrary keyword arguments. Subclass of :class:`set` can now define a ``__new__()`` method with additional keyword parameters without overriding also ``__init__()``. @@ -1317,7 +1317,7 @@ expressions. .. nonce: Kp5FxD .. section: Core and Builtins -Corrent the syntax error message regarding multiple exception types to not +Correct the syntax error message regarding multiple exception types to not refer to "exception groups". Patch by Pablo Galindo .. @@ -3068,8 +3068,8 @@ prevented parallel computation as other :mod:`hashlib` algorithms support. .. section: Library It's now possible to receive the type of service (ToS), a.k.a. -differentiated services (DS), a.k.a. differenciated services code point -(DSCP) and excplicit congestion notification (ECN) IP header fields with +differentiated services (DS), a.k.a. differentiated services code point +(DSCP) and explicit congestion notification (ECN) IP header fields with ``socket.IP_RECVTOS``. .. @@ -3242,7 +3242,7 @@ Patch by Erlend E. Aasland. AIX: `Lib/_aix_support.get_platform()` may fail in an AIX WPAR. The fileset bos.rte appears to have a builddate in both LPAR and WPAR so this fileset is -queried rather than bos.mp64. To prevent a similiar situation (no builddate +queried rather than bos.mp64. To prevent a similar situation (no builddate in ODM) a value (9988) sufficient for completing a build is provided. Patch by M Felt. @@ -3717,7 +3717,7 @@ RFC. .. nonce: zMFGMV .. section: Documentation -Replaced occurences of uppercase "Web" and "Internet" with lowercase +Replaced occurrences of uppercase "Web" and "Internet" with lowercase versions per the 2016 revised Associated Press Style Book. .. diff --git a/Misc/NEWS.d/3.9.0a1.rst b/Misc/NEWS.d/3.9.0a1.rst index 0a6a6eb287145..a9b6694c133f1 100644 --- a/Misc/NEWS.d/3.9.0a1.rst +++ b/Misc/NEWS.d/3.9.0a1.rst @@ -279,7 +279,7 @@ visited by ``tp_traverse()`` are valid. .. nonce: Xgc6F_ .. section: Core and Builtins -Remove unecessary intersection and update set operation in dictview with +Remove unnecessary intersection and update set operation in dictview with empty set. (Contributed by Dong-hee Na in :issue:`38210`.) .. @@ -868,7 +868,7 @@ Fix the :c:func:`PySys_Audit` call in :class:`mmap.mmap`. .. nonce: WJkgKV .. section: Core and Builtins -Remove an unnecssary Py_XINCREF in classobject.c. +Remove an unnecessary Py_XINCREF in classobject.c. .. @@ -1224,7 +1224,7 @@ Anthony Sottile. .. nonce: cyq5nr .. section: Library -Now :func:`~logging.config.fileConfig` correcty sets the .name of handlers +Now :func:`~logging.config.fileConfig` correctly sets the .name of handlers loaded. .. @@ -1637,7 +1637,7 @@ Preserve subclassing in inspect.Signature.from_callable. .. nonce: nHAbuJ .. section: Library -Names of hashing algorithms frome OpenSSL are now normalized to follow +Names of hashing algorithms from OpenSSL are now normalized to follow Python's naming conventions. For example OpenSSL uses sha3-512 instead of sha3_512 or blake2b512 instead of blake2b. @@ -2403,7 +2403,7 @@ Fixed comparisons of :class:`datetime.timedelta` and .. nonce: 7UV5d0 .. section: Library -Syncronize ``importlib.metadata`` with `importlib_metadata 0.19 +Synchronize ``importlib.metadata`` with `importlib_metadata 0.19 `_, improving handling of EGG-INFO files and fixing a crash when entry point names contained colons. @@ -3002,7 +3002,7 @@ on platforms with 16-bit :c:type:`wchar_t` (for example, Windows and AIX). In a subinterpreter, spawning a daemon thread now raises an exception. Daemon threads were never supported in subinterpreters. Previously, the -subinterpreter finalization crashed with a Pyton fatal error if a daemon +subinterpreter finalization crashed with a Python fatal error if a daemon thread was still running. .. @@ -3065,7 +3065,7 @@ internal tasks weak set is changed by another thread during iteration. .. section: Library :class:`_pyio.IOBase` destructor now does nothing if getting the ``closed`` -attribute fails to better mimick :class:`_io.IOBase` finalizer. +attribute fails to better mimic :class:`_io.IOBase` finalizer. .. @@ -4038,7 +4038,7 @@ crypto policies. Use PKCS#8 format with AES256 encryption instead. .. nonce: _x-9uH .. section: Tests -test.support now has a helper function to check for availibility of a hash +test.support now has a helper function to check for availability of a hash digest function. Several tests are refactored avoid MD5 and use SHA256 instead. Other tests are marked to use MD5 and skipped when MD5 is disabled. diff --git a/Misc/NEWS.d/3.9.0a3.rst b/Misc/NEWS.d/3.9.0a3.rst index 6c71d7e839d05..77ccc7453c215 100644 --- a/Misc/NEWS.d/3.9.0a3.rst +++ b/Misc/NEWS.d/3.9.0a3.rst @@ -178,7 +178,7 @@ last iteration of asynchronous for loops. Patch by Pablo Galindo. .. nonce: WG9alt .. section: Core and Builtins -Fix incorrent line execution reporting in trace functions when tracing +Fix incorrect line execution reporting in trace functions when tracing exception handlers with name binding. Patch by Pablo Galindo. .. @@ -685,7 +685,7 @@ but not required. Patch by Juergen Gmach. .. section: Library Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result -upon inheritence. Patch by Bar Harel. +upon inheritance. Patch by Bar Harel. .. diff --git a/Misc/NEWS.d/3.9.0a5.rst b/Misc/NEWS.d/3.9.0a5.rst index 355a3fc22350c..49a118ad7e430 100644 --- a/Misc/NEWS.d/3.9.0a5.rst +++ b/Misc/NEWS.d/3.9.0a5.rst @@ -175,7 +175,7 @@ convention. Patch by Dong-hee Na. .. nonce: FE9S21 .. section: Core and Builtins -Chaged list overallocation strategy. It no longer overallocates if the new +Changed list overallocation strategy. It no longer overallocates if the new size is closer to overallocated size than to the old size and adds padding. .. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 72fa426cd79b9..8e79f52130622 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -285,7 +285,7 @@ macro Py_UNBLOCK_THREADS macro Py_END_ALLOW_THREADS added 3.2 -# The following were added in PC/python3.def in the intial stable ABI commit, +# The following were added in PC/python3.def in the initial stable ABI commit, # 4d0d471a8031de90a2b1ce99c4ac4780e60b3bc9, # and later amendments in 3.2: # 0d012f284be829c6217f60523db0e1671b7db9d9 From webhook-mailer at python.org Wed Oct 6 13:49:48 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 17:49:48 -0000 Subject: [Python-checkins] bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) Message-ID: https://github.com/python/cpython/commit/0571b934f5f9198c3461a7b631d7073ac0a5676f commit: 0571b934f5f9198c3461a7b631d7073ac0a5676f branch: main author: rtobar committer: ambv date: 2021-10-06T19:49:44+02:00 summary: bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) Operating systems without support for TCP_NODELAY will raise an OSError when trying to set the socket option, but the show can still go on. files: A Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst M Lib/http/client.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 08cf2ed9b3716..a6ab135b2c387 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -70,6 +70,7 @@ import email.parser import email.message +import errno import http import io import re @@ -939,7 +940,12 @@ def connect(self): sys.audit("http.client.connect", self, self.host, self.port) self.sock = self._create_connection( (self.host,self.port), self.timeout, self.source_address) - self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + # Might fail in OSs that don't implement TCP_NODELAY + try: + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + except OSError as e: + if e.errno != errno.ENOPROTOOPT: + raise if self._tunnel_host: self._tunnel() diff --git a/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst new file mode 100644 index 0000000000000..eeb49310e8f66 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst @@ -0,0 +1 @@ +Fixed :class:`http.client.HTTPConnection` to work properly in OSs that don't support the ``TCP_NODELAY`` socket option. From webhook-mailer at python.org Wed Oct 6 13:55:25 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 06 Oct 2021 17:55:25 -0000 Subject: [Python-checkins] Fix typos in the Tools directory (GH-28769) Message-ID: https://github.com/python/cpython/commit/682aecfdeba481c876bfc9f3796c635bd5b5df50 commit: 682aecfdeba481c876bfc9f3796c635bd5b5df50 branch: main author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-06T10:55:16-07:00 summary: Fix typos in the Tools directory (GH-28769) Like #28744 but for the Tools directory. [skip issue] Opening a related issue is pending python/psf-infra-meta#130 Automerge-Triggered-By: GH:pablogsal files: M Tools/c-analyzer/c_analyzer/__main__.py M Tools/c-analyzer/c_analyzer/info.py M Tools/c-analyzer/c_common/logging.py M Tools/c-analyzer/c_common/strutil.py M Tools/c-analyzer/c_parser/info.py M Tools/c-analyzer/c_parser/parser/__init__.py M Tools/c-analyzer/c_parser/preprocessor/__init__.py M Tools/c-analyzer/cpython/__main__.py M Tools/c-analyzer/cpython/ignored.tsv M Tools/peg_generator/pegen/c_generator.py M Tools/peg_generator/pegen/first_sets.py M Tools/peg_generator/scripts/download_pypi_packages.py M Tools/pynche/ColorDB.py M Tools/scripts/dutree.doc M Tools/scripts/stable_abi.py diff --git a/Tools/c-analyzer/c_analyzer/__main__.py b/Tools/c-analyzer/c_analyzer/__main__.py index 24fc6cd182656..5d89b29adf899 100644 --- a/Tools/c-analyzer/c_analyzer/__main__.py +++ b/Tools/c-analyzer/c_analyzer/__main__.py @@ -482,7 +482,7 @@ def cmd_data(datacmd, filenames, known=None, *, cmd_analyze, ), 'data': ( - 'check/manage local data (e.g. knwon types, ignored vars, caches)', + 'check/manage local data (e.g. known types, ignored vars, caches)', [_cli_data], cmd_data, ), diff --git a/Tools/c-analyzer/c_analyzer/info.py b/Tools/c-analyzer/c_analyzer/info.py index b75918e5e7a68..27c3a5a4ee76f 100644 --- a/Tools/c-analyzer/c_analyzer/info.py +++ b/Tools/c-analyzer/c_analyzer/info.py @@ -230,11 +230,11 @@ def fix_filename(self, relroot=fsutil.USE_CWD, **kwargs): return self def as_rowdata(self, columns=None): - # XXX finsih! + # XXX finish! return self.item.as_rowdata(columns) def render_rowdata(self, columns=None): - # XXX finsih! + # XXX finish! return self.item.render_rowdata(columns) def render(self, fmt='line', *, itemonly=False): diff --git a/Tools/c-analyzer/c_common/logging.py b/Tools/c-analyzer/c_common/logging.py index 12398f7e385fd..10af852ec3c5e 100644 --- a/Tools/c-analyzer/c_common/logging.py +++ b/Tools/c-analyzer/c_common/logging.py @@ -41,7 +41,7 @@ def configure_logger(logger, verbosity=VERBOSITY, *, def hide_emit_errors(): """Ignore errors while emitting log entries. - Rather than printing a message desribing the error, we show nothing. + Rather than printing a message describing the error, we show nothing. """ # For now we simply ignore all exceptions. If we wanted to ignore # specific ones (e.g. BrokenPipeError) then we would need to use diff --git a/Tools/c-analyzer/c_common/strutil.py b/Tools/c-analyzer/c_common/strutil.py index e7535d45bbba2..07193c091e4c3 100644 --- a/Tools/c-analyzer/c_common/strutil.py +++ b/Tools/c-analyzer/c_common/strutil.py @@ -26,7 +26,7 @@ def parse_entries(entries, *, ignoresep=None): # We read the entire file here to ensure the file # gets closed sooner rather than later. Note that # the file would stay open if this iterator is never - # exchausted. + # exhausted. lines = infile.read().splitlines() for line in _iter_significant_lines(lines): yield line, filename diff --git a/Tools/c-analyzer/c_parser/info.py b/Tools/c-analyzer/c_parser/info.py index 98ff511cfe64a..697b1f26dc215 100644 --- a/Tools/c-analyzer/c_parser/info.py +++ b/Tools/c-analyzer/c_parser/info.py @@ -1029,7 +1029,7 @@ def _resolve_data(cls, data): @classmethod def _raw_data(self, data): - # XXX finsh! + # XXX finish! return data @classmethod @@ -1255,7 +1255,7 @@ def _resolve_data(cls, data): @classmethod def _raw_data(self, data): - # XXX finsih! + # XXX finish! return data @classmethod @@ -1296,12 +1296,12 @@ class Statement(HighlevelParsedItem): @classmethod def _resolve_data(cls, data): - # XXX finsih! + # XXX finish! return data, None @classmethod def _raw_data(self, data): - # XXX finsih! + # XXX finish! return data @classmethod diff --git a/Tools/c-analyzer/c_parser/parser/__init__.py b/Tools/c-analyzer/c_parser/parser/__init__.py index 39056099f5e91..df70aae66b776 100644 --- a/Tools/c-analyzer/c_parser/parser/__init__.py +++ b/Tools/c-analyzer/c_parser/parser/__init__.py @@ -7,7 +7,7 @@ Furthermore, the grammar rules for the C syntax (particularly as described in the K&R book) actually describe a superset, of which the -full C langage is a proper subset. Here are some of the extra +full C language is a proper subset. Here are some of the extra conditions that must be applied when parsing C code: * ... @@ -90,7 +90,7 @@ * no "inline" type declarations (struct, union, enum) in function parameters ~(including function pointers)~ * no "inline" type decls in function return types -* no superflous parentheses in declarators +* no superfluous parentheses in declarators * var decls in for loops are always "simple" (e.g. no inline types) * only inline struct/union/enum decls may be anonymouns (without a name) * no function pointers in function pointer parameters diff --git a/Tools/c-analyzer/c_parser/preprocessor/__init__.py b/Tools/c-analyzer/c_parser/preprocessor/__init__.py index 8da4d8cadf7d0..e38176fee31fa 100644 --- a/Tools/c-analyzer/c_parser/preprocessor/__init__.py +++ b/Tools/c-analyzer/c_parser/preprocessor/__init__.py @@ -19,7 +19,7 @@ logger = logging.getLogger(__name__) -# Supprted "source": +# Supported "source": # * filename (string) # * lines (iterable) # * text (string) @@ -156,7 +156,7 @@ def handling_errors(ignore_exc=None, *, log_err=None): # tools _COMPILERS = { - # matching disutils.ccompiler.compiler_class: + # matching distutils.ccompiler.compiler_class: 'unix': _gcc.preprocess, 'msvc': None, 'cygwin': None, diff --git a/Tools/c-analyzer/cpython/__main__.py b/Tools/c-analyzer/cpython/__main__.py index a11b687214d2f..06ec871ba75e3 100644 --- a/Tools/c-analyzer/cpython/__main__.py +++ b/Tools/c-analyzer/cpython/__main__.py @@ -342,7 +342,7 @@ def cmd_capi(filenames=None, *, cmd_parse, ), 'data': ( - 'check/manage local data (e.g. knwon types, ignored vars, caches)', + 'check/manage local data (e.g. known types, ignored vars, caches)', [_cli_data], cmd_data, ), diff --git a/Tools/c-analyzer/cpython/ignored.tsv b/Tools/c-analyzer/cpython/ignored.tsv index 302f10cb2bc46..f3fdf3b0d05d9 100644 --- a/Tools/c-analyzer/cpython/ignored.tsv +++ b/Tools/c-analyzer/cpython/ignored.tsv @@ -2106,7 +2106,7 @@ Python/import.c import_find_and_load header - #----------------------- # runtime state -# (look at the bottome of the file) +# (look at the bottom of the file) #----------------------- # modules diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index d15e91098dfe9..29c310fa76b17 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -737,7 +737,7 @@ def handle_alt_normal(self, node: Alt, is_gather: bool, rulename: Optional[str]) self.print( f'D(fprintf(stderr, "%*c+ {rulename}[%d-%d]: %s succeeded!\\n", p->level, \' \', _mark, p->mark, "{node_str}"));' ) - # Prepare to emmit the rule action and do so + # Prepare to emit the rule action and do so if node.action and "EXTRA" in node.action: self._set_up_token_end_metadata_extraction() if self.skip_actions: diff --git a/Tools/peg_generator/pegen/first_sets.py b/Tools/peg_generator/pegen/first_sets.py index 611ef514d09bd..6d794ffa4bfa8 100755 --- a/Tools/peg_generator/pegen/first_sets.py +++ b/Tools/peg_generator/pegen/first_sets.py @@ -56,7 +56,7 @@ def visit_Alt(self, item: Alt) -> Set[str]: result -= to_remove # If the set of new terminals can start with the empty string, - # it means that the item is completelly nullable and we should + # it means that the item is completely nullable and we should # also considering at least the next item in case the current # one fails to parse. diff --git a/Tools/peg_generator/scripts/download_pypi_packages.py b/Tools/peg_generator/scripts/download_pypi_packages.py index 0af876c3ecad4..180309d0bce62 100755 --- a/Tools/peg_generator/scripts/download_pypi_packages.py +++ b/Tools/peg_generator/scripts/download_pypi_packages.py @@ -73,7 +73,7 @@ def main() -> None: package_json = load_json(package_name) try: - print(f"Dowloading and compressing package {package_name} ... ", end="") + print(f"Downloading and compressing package {package_name} ... ", end="") download_package_code(package_name, package_json) print("Done") except (IndexError, KeyError): diff --git a/Tools/pynche/ColorDB.py b/Tools/pynche/ColorDB.py index eb76d4042d33b..c013a60896908 100644 --- a/Tools/pynche/ColorDB.py +++ b/Tools/pynche/ColorDB.py @@ -9,7 +9,7 @@ trouble reading the file, None is returned. You can pass get_colordb() an optional filetype argument. -Supporte file types are: +Supported file types are: X_RGB_TXT -- X Consortium rgb.txt format files. Three columns of numbers from 0 .. 255 separated by whitespace. Arbitrary trailing diff --git a/Tools/scripts/dutree.doc b/Tools/scripts/dutree.doc index 97bd2e2e47cae..490126b0182d1 100644 --- a/Tools/scripts/dutree.doc +++ b/Tools/scripts/dutree.doc @@ -15,7 +15,7 @@ From the keyboard of flee at cs.psu.edu (Felix Lee): :And Perl is definitely awkward with data types. I haven't yet found a :pleasant way of shoving non-trivial data types into Perl's grammar. -Yes, it's pretty aweful at that, alright. Sometimes I write perl programs +Yes, it's pretty awful at that, alright. Sometimes I write perl programs that need them, and sometimes it just takes a little creativity. But sometimes it's not worth it. I actually wrote a C program the other day (gasp) because I didn't want to deal with a game matrix with six links per node. diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index b7fd2c8583ba7..6d7034090f881 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -67,7 +67,7 @@ class Manifest: def add(self, item): if item.name in self.contents: # We assume that stable ABI items do not share names, - # even if they're diferent kinds (e.g. function vs. macro). + # even if they're different kinds (e.g. function vs. macro). raise ValueError(f'duplicate ABI item {item.name}') self.contents[item.name] = item @@ -295,7 +295,7 @@ def do_unixy_check(manifest, args): present_macros = gcc_get_limited_api_macros(['Include/Python.h']) feature_defines = manifest.feature_defines & present_macros - # Check that we have all neded macros + # Check that we have all needed macros expected_macros = set( item.name for item in manifest.select({'macro'}) ) @@ -412,7 +412,7 @@ def binutils_check_library(manifest, library, expected_symbols, dynamic): def gcc_get_limited_api_macros(headers): """Get all limited API macros from headers. - Runs the preprocesor over all the header files in "Include" setting + Runs the preprocessor over all the header files in "Include" setting "-DPy_LIMITED_API" to the correct value for the running version of the interpreter and extracting all macro definitions (via adding -dM to the compiler arguments). @@ -449,7 +449,7 @@ def gcc_get_limited_api_macros(headers): def gcc_get_limited_api_definitions(headers): """Get all limited API definitions from headers. - Run the preprocesor over all the header files in "Include" setting + Run the preprocessor over all the header files in "Include" setting "-DPy_LIMITED_API" to the correct value for the running version of the interpreter. From webhook-mailer at python.org Wed Oct 6 14:23:12 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 18:23:12 -0000 Subject: [Python-checkins] [doc] Mention __slots__ behavior in weakref.rst (GH-21061) Message-ID: https://github.com/python/cpython/commit/b24b47e64355224c1bf4e46ed7c4d9f7df4e6f09 commit: b24b47e64355224c1bf4e46ed7c4d9f7df4e6f09 branch: main author: Jakub Stasiak committer: ambv date: 2021-10-06T20:23:02+02:00 summary: [doc] Mention __slots__ behavior in weakref.rst (GH-21061) It took me longer than I expected to figure out why a random class I dealt with didn't support weak references. I believe this addition will make the __slots__/weakref interaction more discoverable to people having troubles with this. (Before this patch __slots__ was not mentioned in weakref documentation even once). Co-authored-by: ?ukasz Langa files: M Doc/library/weakref.rst diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index 5a8df4c27dc37..1102c634edaf3 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -88,6 +88,10 @@ support weak references but can add support through subclassing:: Extension types can easily be made to support weak references; see :ref:`weakref-support`. +When ``__slots__`` are defined for a given type, weak reference support is +disabled unless a ``'__weakref__'`` string is also present in the sequence of +strings in the ``__slots__`` declaration. +See :ref:`__slots__ documentation ` for details. .. class:: ref(object[, callback]) From webhook-mailer at python.org Wed Oct 6 14:29:34 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 18:29:34 -0000 Subject: [Python-checkins] bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) (GH-28771) Message-ID: https://github.com/python/cpython/commit/4c35a2aa80d7f55573d83651883d8733fac01e31 commit: 4c35a2aa80d7f55573d83651883d8733fac01e31 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T20:29:23+02:00 summary: bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) (GH-28771) Operating systems without support for TCP_NODELAY will raise an OSError when trying to set the socket option, but the show can still go on. (cherry picked from commit 0571b934f5f9198c3461a7b631d7073ac0a5676f) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst M Lib/http/client.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 08cf2ed9b3716..a6ab135b2c387 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -70,6 +70,7 @@ import email.parser import email.message +import errno import http import io import re @@ -939,7 +940,12 @@ def connect(self): sys.audit("http.client.connect", self, self.host, self.port) self.sock = self._create_connection( (self.host,self.port), self.timeout, self.source_address) - self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + # Might fail in OSs that don't implement TCP_NODELAY + try: + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + except OSError as e: + if e.errno != errno.ENOPROTOOPT: + raise if self._tunnel_host: self._tunnel() diff --git a/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst new file mode 100644 index 0000000000000..eeb49310e8f66 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst @@ -0,0 +1 @@ +Fixed :class:`http.client.HTTPConnection` to work properly in OSs that don't support the ``TCP_NODELAY`` socket option. From webhook-mailer at python.org Wed Oct 6 14:29:46 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 18:29:46 -0000 Subject: [Python-checkins] bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) (GH-28770) Message-ID: https://github.com/python/cpython/commit/92018a08240308c5beef9ccc712bef5c2e582926 commit: 92018a08240308c5beef9ccc712bef5c2e582926 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T20:29:41+02:00 summary: bpo-45328: Avoid failure in OSs without TCP_NODELAY support (GH-28646) (GH-28770) Operating systems without support for TCP_NODELAY will raise an OSError when trying to set the socket option, but the show can still go on. (cherry picked from commit 0571b934f5f9198c3461a7b631d7073ac0a5676f) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst M Lib/http/client.py diff --git a/Lib/http/client.py b/Lib/http/client.py index 0fd9021b4a785..a98432e568506 100644 --- a/Lib/http/client.py +++ b/Lib/http/client.py @@ -70,6 +70,7 @@ import email.parser import email.message +import errno import http import io import re @@ -944,7 +945,12 @@ def connect(self): """Connect to the host and port specified in __init__.""" self.sock = self._create_connection( (self.host,self.port), self.timeout, self.source_address) - self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + # Might fail in OSs that don't implement TCP_NODELAY + try: + self.sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1) + except OSError as e: + if e.errno != errno.ENOPROTOOPT: + raise if self._tunnel_host: self._tunnel() diff --git a/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst new file mode 100644 index 0000000000000..eeb49310e8f66 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-30-08-22-44.bpo-45328.8Z-Q0B.rst @@ -0,0 +1 @@ +Fixed :class:`http.client.HTTPConnection` to work properly in OSs that don't support the ``TCP_NODELAY`` socket option. From webhook-mailer at python.org Wed Oct 6 14:32:47 2021 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Oct 2021 18:32:47 -0000 Subject: [Python-checkins] Fix typos in the Include directory (GH-28745) Message-ID: https://github.com/python/cpython/commit/8e8f7522171ef82f2f5049940f815e00e38c6f42 commit: 8e8f7522171ef82f2f5049940f815e00e38c6f42 branch: main author: Christian Clauss committer: gpshead date: 2021-10-06T11:32:38-07:00 summary: Fix typos in the Include directory (GH-28745) files: M Include/abstract.h M Include/cpython/dictobject.h M Include/cpython/pystate.h M Include/cpython/pytime.h M Include/internal/pycore_frame.h M Include/internal/pycore_traceback.h M Include/object.h diff --git a/Include/abstract.h b/Include/abstract.h index e8d3f9293b98d..9eaab6b2e054c 100644 --- a/Include/abstract.h +++ b/Include/abstract.h @@ -318,7 +318,7 @@ PyAPI_FUNC(int) PyObject_DelItem(PyObject *o, PyObject *key); /* Takes an arbitrary object which must support the (character, single segment) buffer interface and returns a pointer to a read-only memory location - useable as character based input for subsequent processing. + usable as character based input for subsequent processing. Return 0 on success. buffer and buffer_len are only set in case no error occurs. Otherwise, -1 is returned and an exception set. */ diff --git a/Include/cpython/dictobject.h b/Include/cpython/dictobject.h index ba118788f7b1b..e97969be4de0c 100644 --- a/Include/cpython/dictobject.h +++ b/Include/cpython/dictobject.h @@ -23,7 +23,7 @@ typedef struct { /* If ma_values is NULL, the table is "combined": keys and values are stored in ma_keys. - If ma_values is not NULL, the table is splitted: + If ma_values is not NULL, the table is split: keys are stored in ma_keys and values are stored in ma_values */ PyDictValues *ma_values; } PyDictObject; diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index ab4bf8bf8483c..ca0de87ab0e5c 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -263,7 +263,7 @@ PyAPI_FUNC(int) _PyInterpreterState_GetConfigCopy( PyAPI_FUNC(int) _PyInterpreterState_SetConfig( const struct PyConfig *config); -// Get the configuration of the currrent interpreter. +// Get the configuration of the current interpreter. // The caller must hold the GIL. PyAPI_FUNC(const PyConfig*) _Py_GetConfig(void); diff --git a/Include/cpython/pytime.h b/Include/cpython/pytime.h index f32148aa8448d..23d4f16a8fd84 100644 --- a/Include/cpython/pytime.h +++ b/Include/cpython/pytime.h @@ -134,13 +134,13 @@ PyAPI_FUNC(_PyTime_t) _PyTime_FromNanoseconds(_PyTime_t ns); PyAPI_FUNC(int) _PyTime_FromNanosecondsObject(_PyTime_t *t, PyObject *obj); -/* Convert a number of seconds (Python float or int) to a timetamp. +/* Convert a number of seconds (Python float or int) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromSecondsObject(_PyTime_t *t, PyObject *obj, _PyTime_round_t round); -/* Convert a number of milliseconds (Python float or int, 10^-3) to a timetamp. +/* Convert a number of milliseconds (Python float or int, 10^-3) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromMillisecondsObject(_PyTime_t *t, PyObject *obj, diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 6afb95c3ad62e..489e115a8645c 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -128,7 +128,7 @@ _PyFrame_GetFrameObject(InterpreterFrame *frame) /* Clears all references in the frame. * If take is non-zero, then the InterpreterFrame frame - * may be transfered to the frame object it references + * may be transferred to the frame object it references * instead of being cleared. Either way * the caller no longer owns the references * in the frame. diff --git a/Include/internal/pycore_traceback.h b/Include/internal/pycore_traceback.h index 4d282308769dc..c01a47639d5e3 100644 --- a/Include/internal/pycore_traceback.h +++ b/Include/internal/pycore_traceback.h @@ -51,7 +51,7 @@ PyAPI_FUNC(void) _Py_DumpTraceback( _PyGILState_GetInterpreterStateUnsafe() in last resort. It is better to pass NULL to interp and current_tstate, the function tries - different options to retrieve these informations. + different options to retrieve this information. This function is signal safe. */ diff --git a/Include/object.h b/Include/object.h index 490cbffd5f107..c3062cb01c8e0 100644 --- a/Include/object.h +++ b/Include/object.h @@ -600,7 +600,7 @@ static inline PyObject* _Py_XNewRef(PyObject *obj) } // Py_NewRef() and Py_XNewRef() are exported as functions for the stable ABI. -// Names overriden with macros by static inline functions for best +// Names overridden with macros by static inline functions for best // performances. #define Py_NewRef(obj) _Py_NewRef(_PyObject_CAST(obj)) #define Py_XNewRef(obj) _Py_XNewRef(_PyObject_CAST(obj)) From webhook-mailer at python.org Wed Oct 6 14:52:32 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 18:52:32 -0000 Subject: [Python-checkins] [doc] Mention __slots__ behavior in weakref.rst (GH-21061) (GH-28772) Message-ID: https://github.com/python/cpython/commit/ce121fd8755d4db9511ce4aab39d0577165e118e commit: ce121fd8755d4db9511ce4aab39d0577165e118e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T20:52:22+02:00 summary: [doc] Mention __slots__ behavior in weakref.rst (GH-21061) (GH-28772) It took me longer than I expected to figure out why a random class I dealt with didn't support weak references. I believe this addition will make the __slots__/weakref interaction more discoverable to people having troubles with this. (Before this patch __slots__ was not mentioned in weakref documentation even once). Co-authored-by: ?ukasz Langa (cherry picked from commit b24b47e64355224c1bf4e46ed7c4d9f7df4e6f09) Co-authored-by: Jakub Stasiak files: M Doc/library/weakref.rst diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index b88543e445372..4b0945c020f84 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -88,6 +88,10 @@ support weak references but can add support through subclassing:: Extension types can easily be made to support weak references; see :ref:`weakref-support`. +When ``__slots__`` are defined for a given type, weak reference support is +disabled unless a ``'__weakref__'`` string is also present in the sequence of +strings in the ``__slots__`` declaration. +See :ref:`__slots__ documentation ` for details. .. class:: ref(object[, callback]) From webhook-mailer at python.org Wed Oct 6 14:52:53 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 06 Oct 2021 18:52:53 -0000 Subject: [Python-checkins] [doc] Mention __slots__ behavior in weakref.rst (GH-21061) (GH-28773) Message-ID: https://github.com/python/cpython/commit/4f161e65a011f287227c944fad9987446644041f commit: 4f161e65a011f287227c944fad9987446644041f branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-06T20:52:48+02:00 summary: [doc] Mention __slots__ behavior in weakref.rst (GH-21061) (GH-28773) It took me longer than I expected to figure out why a random class I dealt with didn't support weak references. I believe this addition will make the __slots__/weakref interaction more discoverable to people having troubles with this. (Before this patch __slots__ was not mentioned in weakref documentation even once). Co-authored-by: ?ukasz Langa (cherry picked from commit b24b47e64355224c1bf4e46ed7c4d9f7df4e6f09) Co-authored-by: Jakub Stasiak files: M Doc/library/weakref.rst diff --git a/Doc/library/weakref.rst b/Doc/library/weakref.rst index b88543e445372..4b0945c020f84 100644 --- a/Doc/library/weakref.rst +++ b/Doc/library/weakref.rst @@ -88,6 +88,10 @@ support weak references but can add support through subclassing:: Extension types can easily be made to support weak references; see :ref:`weakref-support`. +When ``__slots__`` are defined for a given type, weak reference support is +disabled unless a ``'__weakref__'`` string is also present in the sequence of +strings in the ``__slots__`` declaration. +See :ref:`__slots__ documentation ` for details. .. class:: ref(object[, callback]) From webhook-mailer at python.org Wed Oct 6 15:09:42 2021 From: webhook-mailer at python.org (zooba) Date: Wed, 06 Oct 2021 19:09:42 -0000 Subject: [Python-checkins] bpo-45375: Fix off by one error in buffer allocation (GH-28764) Message-ID: https://github.com/python/cpython/commit/6c942a86a4fb4c8b731cb1bd2933dba554eb79cd commit: 6c942a86a4fb4c8b731cb1bd2933dba554eb79cd branch: main author: Steve Dower committer: zooba date: 2021-10-06T20:09:33+01:00 summary: bpo-45375: Fix off by one error in buffer allocation (GH-28764) files: M PC/getpathp.c diff --git a/PC/getpathp.c b/PC/getpathp.c index 98a754976c670..062697b3e9afd 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -266,7 +266,7 @@ canonicalize(wchar_t *buffer, const wchar_t *path) } if (PathIsRelativeW(path)) { - wchar_t buff[MAXPATHLEN]; + wchar_t buff[MAXPATHLEN + 1]; if (!GetCurrentDirectoryW(MAXPATHLEN, buff)) { return _PyStatus_ERR("unable to find current working directory"); } From webhook-mailer at python.org Wed Oct 6 18:55:31 2021 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Oct 2021 22:55:31 -0000 Subject: [Python-checkins] Fix typos in the Python directory (GH-28767) Message-ID: https://github.com/python/cpython/commit/db693df3e112c5a61f2cbef63eedce3a36520ded commit: db693df3e112c5a61f2cbef63eedce3a36520ded branch: main author: Christian Clauss committer: gpshead date: 2021-10-06T15:55:27-07:00 summary: Fix typos in the Python directory (GH-28767) files: M Python/compile.c M Python/fileutils.c M Python/import.c M Python/initconfig.c M Python/pathconfig.c M Python/pythonrun.c M Python/specialize.c diff --git a/Python/compile.c b/Python/compile.c index 2d82d6a1e5a91..0c025acec1491 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -6335,7 +6335,7 @@ compiler_pattern_or(struct compiler *c, pattern_ty p, pattern_context *pc) // cases, though. assert(istores < icontrol); Py_ssize_t rotations = istores + 1; - // Perfom the same rotation on pc->stores: + // Perform the same rotation on pc->stores: PyObject *rotated = PyList_GetSlice(pc->stores, 0, rotations); if (rotated == NULL || @@ -7326,7 +7326,7 @@ consts_dict_keys_inorder(PyObject *dict) return NULL; while (PyDict_Next(dict, &pos, &k, &v)) { i = PyLong_AS_LONG(v); - /* The keys of the dictionary can be tuples wrapping a contant. + /* The keys of the dictionary can be tuples wrapping a constant. * (see compiler_add_o and _PyCode_ConstantKey). In that case * the object we want is always second. */ if (PyTuple_CheckExact(k)) { diff --git a/Python/fileutils.c b/Python/fileutils.c index ecfdc5758eb84..a36415946e218 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -221,7 +221,7 @@ check_force_ascii(void) ch = (unsigned char)0xA7; res = _Py_mbstowcs(&wch, (char*)&ch, 1); if (res != DECODE_ERROR && wch == L'\xA7') { - /* On HP-UX withe C locale or the POSIX locale, + /* On HP-UX with C locale or the POSIX locale, nl_langinfo(CODESET) announces "roman8", whereas mbstowcs() uses Latin1 encoding in practice. Force ASCII in this case. diff --git a/Python/import.c b/Python/import.c index a6170a39c7fdb..731f0f59118d6 100644 --- a/Python/import.c +++ b/Python/import.c @@ -2458,7 +2458,7 @@ _PyImport_BootstrapImp(PyThreadState *tstate) // Mock a ModuleSpec object just good enough for PyModule_FromDefAndSpec(): // an object with just a name attribute. // - // _imp.__spec__ is overriden by importlib._bootstrap._instal() anyway. + // _imp.__spec__ is overridden by importlib._bootstrap._instal() anyway. PyObject *attrs = Py_BuildValue("{sO}", "name", name); if (attrs == NULL) { goto error; diff --git a/Python/initconfig.c b/Python/initconfig.c index 9fa202a7da5c0..2e3cde83b2534 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -98,7 +98,7 @@ static const char usage_3[] = "\ -X no_debug_ranges: disable the inclusion of the tables mapping extra location \n\ information (end line, start column offset and end column offset) to every \n\ instruction in code objects. This is useful when smaller code objects and pyc \n\ - files are desired as well as supressing the extra visual location indicators \n\ + files are desired as well as suppressing the extra visual location indicators \n\ when the interpreter displays tracebacks.\n\ -X frozen_modules=[on|off]: whether or not frozen modules should be used.\n\ The default is \"on\" (or \"off\" if you are running a local build).\n\ @@ -142,7 +142,7 @@ static const char usage_6[] = "PYTHONNODEBUGRANGES: If this variable is set, it disables the inclusion of the \n" " tables mapping extra location information (end line, start column offset \n" " and end column offset) to every instruction in code objects. This is useful \n" -" when smaller cothe de objects and pyc files are desired as well as supressing the \n" +" when smaller cothe de objects and pyc files are desired as well as suppressing the \n" " extra visual location indicators when the interpreter displays tracebacks.\n"; #if defined(MS_WINDOWS) @@ -2549,7 +2549,7 @@ warnoptions_append(PyConfig *config, PyWideStringList *options, { /* config_init_warnoptions() add existing config warnoptions at the end: ensure that the new option is not already present in this list to - prevent change the options order whne config_init_warnoptions() is + prevent change the options order when config_init_warnoptions() is called twice. */ if (_PyWideStringList_Find(&config->warnoptions, option)) { /* Already present: do nothing */ diff --git a/Python/pathconfig.c b/Python/pathconfig.c index d49bd3c854940..2ebb9d39b8c2c 100644 --- a/Python/pathconfig.c +++ b/Python/pathconfig.c @@ -411,7 +411,7 @@ config_init_pathconfig(PyConfig *config, int compute_path_config) #undef COPY_ATTR #ifdef MS_WINDOWS - /* If a ._pth file is found: isolated and site_import are overriden */ + /* If a ._pth file is found: isolated and site_import are overridden */ if (pathconfig.isolated != -1) { config->isolated = pathconfig.isolated; } diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 0e0262c0e8c69..6cecef9793228 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -2,7 +2,7 @@ /* Top level execution of Python code (including in __main__) */ /* To help control the interfaces between the startup, execution and - * shutdown code, the phases are split across separate modules (boostrap, + * shutdown code, the phases are split across separate modules (bootstrap, * pythonrun, shutdown) */ @@ -943,7 +943,7 @@ print_exception(PyObject *f, PyObject *value) if (end_lineno > lineno) { end_offset = (error_line != NULL) ? line_size : -1; } - // Limit the ammount of '^' that we can display to + // Limit the amount of '^' that we can display to // the size of the text in the source line. if (error_line != NULL && end_offset > line_size + 1) { end_offset = line_size + 1; diff --git a/Python/specialize.c b/Python/specialize.c index 1ab79bf3ea0c5..4e025384a6252 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -588,7 +588,7 @@ specialize_dict_access( { assert(kind == NON_OVERRIDING || kind == NON_DESCRIPTOR || kind == ABSENT || kind == BUILTIN_CLASSMETHOD || kind == PYTHON_CLASSMETHOD); - // No desciptor, or non overriding. + // No descriptor, or non overriding. if (type->tp_dictoffset < 0) { SPECIALIZATION_FAIL(base_op, SPEC_FAIL_OUT_OF_RANGE); return 0; From webhook-mailer at python.org Wed Oct 6 19:13:58 2021 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Oct 2021 23:13:58 -0000 Subject: [Python-checkins] Fix typos in the Lib directory (GH-28775) Message-ID: https://github.com/python/cpython/commit/745c9d9dfc1ad6fdfdf1d07420c6273ff67fa5be commit: 745c9d9dfc1ad6fdfdf1d07420c6273ff67fa5be branch: main author: Christian Clauss committer: gpshead date: 2021-10-06T16:13:48-07:00 summary: Fix typos in the Lib directory (GH-28775) Fix typos in the Lib directory as identified by codespell. Co-authored-by: Terry Jan Reedy files: M Lib/asyncio/events.py M Lib/asyncio/unix_events.py M Lib/ctypes/_aix.py M Lib/ctypes/test/test_structures.py M Lib/difflib.py M Lib/distutils/ccompiler.py M Lib/distutils/command/install.py M Lib/email/errors.py M Lib/html/parser.py M Lib/idlelib/ChangeLog M Lib/idlelib/idle_test/htest.py M Lib/idlelib/idle_test/mock_tk.py M Lib/idlelib/idle_test/test_multicall.py M Lib/idlelib/idle_test/test_pyparse.py M Lib/importlib/_adapters.py M Lib/lib2to3/fixes/fix_metaclass.py M Lib/lib2to3/fixes/fix_paren.py M Lib/linecache.py M Lib/pickle.py M Lib/platform.py M Lib/sqlite3/test/test_dbapi.py M Lib/sysconfig.py M Lib/test/datetimetester.py M Lib/test/decimaltestdata/abs.decTest M Lib/test/decimaltestdata/extra.decTest M Lib/test/libregrtest/refleak.py M Lib/test/pickletester.py M Lib/test/support/threading_helper.py M Lib/test/test__xxsubinterpreters.py M Lib/test/test_asyncio/test_streams.py M Lib/test/test_asyncio/test_subprocess.py M Lib/test/test_asyncio/test_tasks.py M Lib/test/test_capi.py M Lib/test/test_collections.py M Lib/test/test_dataclasses.py M Lib/test/test_descr.py M Lib/test/test_dict.py M Lib/test/test_dict_version.py M Lib/test/test_dtrace.py M Lib/test/test_email/test_message.py M Lib/test/test_embed.py M Lib/test/test_exceptions.py M Lib/test/test_future.py M Lib/test/test_lltrace.py M Lib/test/test_locale.py M Lib/test/test_long.py M Lib/test/test_pathlib.py M Lib/test/test_strftime.py M Lib/test/test_sys.py M Lib/test/test_tempfile.py M Lib/test/test_time.py M Lib/test/test_unparse.py M Lib/test/test_weakref.py M Lib/test/test_wsgiref.py M Lib/test/test_xml_etree.py M Lib/test/test_xmlrpc.py M Lib/threading.py M Lib/tkinter/__init__.py M Lib/tkinter/test/test_ttk/test_widgets.py M Lib/unittest/async_case.py M Lib/unittest/test/testmock/testsealable.py M Lib/venv/scripts/common/Activate.ps1 M Lib/wsgiref/validate.py M Lib/zoneinfo/_zoneinfo.py diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index b966ad26bf467..7abaaca2d2b28 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -479,7 +479,7 @@ async def connect_read_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def connect_write_pipe(self, protocol_factory, pipe): @@ -492,7 +492,7 @@ async def connect_write_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def subprocess_shell(self, protocol_factory, cmd, *, diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index e4f445e95026b..4cef914b9fb9e 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1379,7 +1379,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classe requires it + # abstract base classes requires it return True def attach_loop(self, loop): diff --git a/Lib/ctypes/_aix.py b/Lib/ctypes/_aix.py index 26959d90a4dd6..fc3e95cbcc88a 100644 --- a/Lib/ctypes/_aix.py +++ b/Lib/ctypes/_aix.py @@ -163,7 +163,7 @@ def get_legacy(members): return member else: # 32-bit legacy names - both shr.o and shr4.o exist. - # shr.o is the preffered name so we look for shr.o first + # shr.o is the preferred name so we look for shr.o first # i.e., shr4.o is returned only when shr.o does not exist for name in ['shr.o', 'shr4.o']: member = get_one_match(re.escape(name), members) diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py index 245cd94c5cdd9..97ad2b8ed8a50 100644 --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -443,7 +443,7 @@ def __del__(self): s = Test(1, 2, 3) # Test the StructUnionType_paramfunc() code path which copies the - # structure: if the stucture is larger than sizeof(void*). + # structure: if the structure is larger than sizeof(void*). self.assertGreater(sizeof(s), sizeof(c_void_p)) dll = CDLL(_ctypes_test.__file__) @@ -451,7 +451,7 @@ def __del__(self): func.argtypes = (Test,) func.restype = None func(s) - # bpo-37140: Passing the structure by refrence must not call + # bpo-37140: Passing the structure by reference must not call # its finalizer! self.assertEqual(finalizer_calls, []) self.assertEqual(s.first, 1) diff --git a/Lib/difflib.py b/Lib/difflib.py index 480bad2224c8e..afd8a0c7c5b61 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -62,7 +62,7 @@ class SequenceMatcher: notion, pairing up elements that appear uniquely in each sequence. That, and the method here, appear to yield more intuitive difference reports than does diff. This method appears to be the least vulnerable - to synching up on blocks of "junk lines", though (like blank lines in + to syncing up on blocks of "junk lines", though (like blank lines in ordinary text files, or maybe "

" lines in HTML files). That may be because this is the only method of the 3 that has a *concept* of "junk" . diff --git a/Lib/distutils/ccompiler.py b/Lib/distutils/ccompiler.py index b5ef143e72c56..4c47f2ed245d4 100644 --- a/Lib/distutils/ccompiler.py +++ b/Lib/distutils/ccompiler.py @@ -392,7 +392,7 @@ def _fix_compile_args(self, output_dir, macros, include_dirs): return output_dir, macros, include_dirs def _prep_compile(self, sources, output_dir, depends=None): - """Decide which souce files must be recompiled. + """Decide which source files must be recompiled. Determine the list of object files corresponding to 'sources', and figure out which ones really need to be recompiled. diff --git a/Lib/distutils/command/install.py b/Lib/distutils/command/install.py index 26696cfb9dcf9..01d5331a63069 100644 --- a/Lib/distutils/command/install.py +++ b/Lib/distutils/command/install.py @@ -31,7 +31,7 @@ # while making the sysconfig module the single point of truth. # This makes it easier for OS distributions where they need to # alter locations for packages installations in a single place. -# Note that this module is depracated (PEP 632); all consumers +# Note that this module is deprecated (PEP 632); all consumers # of this information should switch to using sysconfig directly. INSTALL_SCHEMES = {"unix_prefix": {}, "unix_home": {}, "nt": {}} @@ -43,7 +43,7 @@ sys_key = key sys_scheme = sysconfig._INSTALL_SCHEMES[sys_scheme_name] if key == "headers" and key not in sys_scheme: - # On POSIX-y platofrms, Python will: + # On POSIX-y platforms, Python will: # - Build from .h files in 'headers' (only there when # building CPython) # - Install .h files to 'include' diff --git a/Lib/email/errors.py b/Lib/email/errors.py index 1d258c34fc9d4..3ad0056554996 100644 --- a/Lib/email/errors.py +++ b/Lib/email/errors.py @@ -110,4 +110,4 @@ class NonASCIILocalPartDefect(HeaderDefect): # parsing messages decoded from binary. class InvalidDateDefect(HeaderDefect): - """Header has unparseable or invalid date""" + """Header has unparsable or invalid date""" diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 9e49effca1fcc..58f6bb3b1e932 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -405,7 +405,7 @@ def parse_endtag(self, i): tagname = namematch.group(1).lower() # consume and ignore other stuff between the name and the > # Note: this is not 100% correct, since we might have things like - # , but looking for > after tha name should cover + # , but looking for > after the name should cover # most of the cases and is much simpler gtpos = rawdata.find('>', namematch.end()) self.handle_endtag(tagname) diff --git a/Lib/idlelib/ChangeLog b/Lib/idlelib/ChangeLog index d7d7e1efdb1d3..c8960cfa535d0 100644 --- a/Lib/idlelib/ChangeLog +++ b/Lib/idlelib/ChangeLog @@ -1175,7 +1175,7 @@ Wed Mar 10 05:18:02 1999 Guido van Rossum classes in selected module methods of selected class - Sinlge clicking in a directory, module or class item updates the next + Single clicking in a directory, module or class item updates the next column with info about the selected item. Double clicking in a module, class or method item opens the file (and selects the clicked item if it is a class or method). diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py index 1373b7642a6ea..666ff4cb84851 100644 --- a/Lib/idlelib/idle_test/htest.py +++ b/Lib/idlelib/idle_test/htest.py @@ -246,7 +246,7 @@ def _wrapper(parent): # htest # _object_browser_spec = { 'file': 'debugobj', 'kwds': {}, - 'msg': "Double click on items upto the lowest level.\n" + 'msg': "Double click on items up to the lowest level.\n" "Attributes of the objects and related information " "will be displayed side-by-side at each level." } @@ -255,7 +255,7 @@ def _wrapper(parent): # htest # 'file': 'pathbrowser', 'kwds': {}, 'msg': "Test for correct display of all paths in sys.path.\n" - "Toggle nested items upto the lowest level.\n" + "Toggle nested items up to the lowest level.\n" "Double clicking on an item prints a traceback\n" "for an exception that is ignored." } @@ -341,7 +341,7 @@ def _wrapper(parent): # htest # 'file': 'tree', 'kwds': {}, 'msg': "The canvas is scrollable.\n" - "Click on folders upto to the lowest level." + "Click on folders up to to the lowest level." } _undo_delegator_spec = { diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py index db583553838fb..8304734b847a8 100644 --- a/Lib/idlelib/idle_test/mock_tk.py +++ b/Lib/idlelib/idle_test/mock_tk.py @@ -79,7 +79,7 @@ def tearDownClass(cls): --- For 'ask' functions, set func.result return value before calling the method that uses the message function. When messagebox functions are the - only gui alls in a method, this replacement makes the method gui-free, + only GUI calls in a method, this replacement makes the method GUI-free, """ askokcancel = Mbox_func() # True or False askquestion = Mbox_func() # 'yes' or 'no' diff --git a/Lib/idlelib/idle_test/test_multicall.py b/Lib/idlelib/idle_test/test_multicall.py index ba582bb3ca51b..b3a3bfb88f9c3 100644 --- a/Lib/idlelib/idle_test/test_multicall.py +++ b/Lib/idlelib/idle_test/test_multicall.py @@ -37,7 +37,7 @@ def test_init(self): def test_yview(self): # Added for tree.wheel_event - # (it depends on yview to not be overriden) + # (it depends on yview to not be overridden) mc = self.mc self.assertIs(mc.yview, Text.yview) mctext = self.mc(self.root) diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index fb5726db1d821..384db566ac76c 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -284,7 +284,7 @@ def test_get_num_lines_in_stmt(self): tests = ( TestInfo('[x for x in a]\n', 1), # Closed on one line. TestInfo('[x\nfor x in a\n', 2), # Not closed. - TestInfo('[x\\\nfor x in a\\\n', 2), # "", uneeded backslashes. + TestInfo('[x\\\nfor x in a\\\n', 2), # "", unneeded backslashes. TestInfo('[x\nfor x in a\n]\n', 3), # Closed on multi-line. TestInfo('\n"""Docstring comment L1"""\nL2\nL3\nL4\n', 1), TestInfo('\n"""Docstring comment L1\nL2"""\nL3\nL4\n', 1), diff --git a/Lib/importlib/_adapters.py b/Lib/importlib/_adapters.py index 9907b148b396d..ea363d86a564b 100644 --- a/Lib/importlib/_adapters.py +++ b/Lib/importlib/_adapters.py @@ -41,8 +41,8 @@ def _io_wrapper(file, mode='r', *args, **kwargs): class CompatibilityFiles: """ - Adapter for an existing or non-existant resource reader - to provide a compability .files(). + Adapter for an existing or non-existent resource reader + to provide a compatibility .files(). """ class SpecPath(abc.Traversable): @@ -83,7 +83,7 @@ def open(self, mode='r', *args, **kwargs): class ChildPath(abc.Traversable): """ Path tied to a resource reader child. - Can be read but doesn't expose any meaningfull children. + Can be read but doesn't expose any meaningful children. """ def __init__(self, reader, name): diff --git a/Lib/lib2to3/fixes/fix_metaclass.py b/Lib/lib2to3/fixes/fix_metaclass.py index d1cd10d327587..fe547b2228072 100644 --- a/Lib/lib2to3/fixes/fix_metaclass.py +++ b/Lib/lib2to3/fixes/fix_metaclass.py @@ -51,7 +51,7 @@ def fixup_parse_tree(cls_node): # already in the preferred format, do nothing return - # !%@#! oneliners have no suite node, we have to fake one up + # !%@#! one-liners have no suite node, we have to fake one up for i, node in enumerate(cls_node.children): if node.type == token.COLON: break diff --git a/Lib/lib2to3/fixes/fix_paren.py b/Lib/lib2to3/fixes/fix_paren.py index b205aa7e1e93f..df3da5f5232c9 100644 --- a/Lib/lib2to3/fixes/fix_paren.py +++ b/Lib/lib2to3/fixes/fix_paren.py @@ -1,4 +1,4 @@ -"""Fixer that addes parentheses where they are required +"""Fixer that adds parentheses where they are required This converts ``[x for x in 1, 2]`` to ``[x for x in (1, 2)]``.""" diff --git a/Lib/linecache.py b/Lib/linecache.py index 513b17e999880..23191d6501d2a 100644 --- a/Lib/linecache.py +++ b/Lib/linecache.py @@ -154,7 +154,7 @@ def lazycache(filename, module_globals): :return: True if a lazy load is registered in the cache, otherwise False. To register such a load a module loader with a - get_source method must be found, the filename must be a cachable + get_source method must be found, the filename must be a cacheable filename, and the filename must not be already cached. """ if filename in cache: diff --git a/Lib/pickle.py b/Lib/pickle.py index 5ab312f2acaee..e7f30f226101f 100644 --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -1173,7 +1173,7 @@ def __init__(self, file, *, fix_imports=True, used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python 2; these default to 'ASCII' and 'strict', respectively. *encoding* can be - 'bytes' to read theses 8-bit string instances as bytes objects. + 'bytes' to read these 8-bit string instances as bytes objects. """ self._buffers = iter(buffers) if buffers is not None else None self._file_readline = file.readline diff --git a/Lib/platform.py b/Lib/platform.py index 240f701754d4a..9e9b42238fb76 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -1261,7 +1261,7 @@ def platform(aliased=0, terse=0): def _parse_os_release(lines): # These fields are mandatory fields with well-known defaults - # in pratice all Linux distributions override NAME, ID, and PRETTY_NAME. + # in practice all Linux distributions override NAME, ID, and PRETTY_NAME. info = { "NAME": "Linux", "ID": "linux", diff --git a/Lib/sqlite3/test/test_dbapi.py b/Lib/sqlite3/test/test_dbapi.py index 732e21dd3a285..c21688d2a774c 100644 --- a/Lib/sqlite3/test/test_dbapi.py +++ b/Lib/sqlite3/test/test_dbapi.py @@ -616,7 +616,7 @@ def test_fetchone_no_statement(self): self.assertEqual(row, None) def test_array_size(self): - # must default ot 1 + # must default to 1 self.assertEqual(self.cu.arraysize, 1) # now set to 2 diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py index 95b48f6429d5f..daf9f000060a3 100644 --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -184,7 +184,7 @@ def is_python_build(check_home=False): if _PYTHON_BUILD: for scheme in ('posix_prefix', 'posix_home'): - # On POSIX-y platofrms, Python will: + # On POSIX-y platforms, Python will: # - Build from .h files in 'headers' (which is only added to the # scheme when building CPython) # - Install .h files to 'include' diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py index 6414f1ace3fed..9f551d9b9748d 100644 --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -4064,7 +4064,7 @@ def test_even_more_compare(self): self.assertEqual(t1, t1) self.assertEqual(t2, t2) - # Equal afer adjustment. + # Equal after adjustment. t1 = self.theclass(1, 12, 31, 23, 59, tzinfo=FixedOffset(1, "")) t2 = self.theclass(2, 1, 1, 3, 13, tzinfo=FixedOffset(3*60+13+2, "")) self.assertEqual(t1, t2) @@ -4903,7 +4903,7 @@ def test_easy(self): # OTOH, these fail! Don't enable them. The difficulty is that # the edge case tests assume that every hour is representable in # the "utc" class. This is always true for a fixed-offset tzinfo - # class (lke utc_real and utc_fake), but not for Eastern or Central. + # class (like utc_real and utc_fake), but not for Eastern or Central. # For these adjacent DST-aware time zones, the range of time offsets # tested ends up creating hours in the one that aren't representable # in the other. For the same reason, we would see failures in the diff --git a/Lib/test/decimaltestdata/abs.decTest b/Lib/test/decimaltestdata/abs.decTest index 01f73d7766648..569b8fcd84ab6 100644 --- a/Lib/test/decimaltestdata/abs.decTest +++ b/Lib/test/decimaltestdata/abs.decTest @@ -20,7 +20,7 @@ version: 2.59 -- This set of tests primarily tests the existence of the operator. --- Additon, subtraction, rounding, and more overflows are tested +-- Addition, subtraction, rounding, and more overflows are tested -- elsewhere. precision: 9 diff --git a/Lib/test/decimaltestdata/extra.decTest b/Lib/test/decimaltestdata/extra.decTest index b630d8e3f9d45..31291202a35e7 100644 --- a/Lib/test/decimaltestdata/extra.decTest +++ b/Lib/test/decimaltestdata/extra.decTest @@ -156,7 +156,7 @@ extr1302 fma -Inf 0E-456 sNaN148 -> NaN Invalid_operation -- max/min/max_mag/min_mag bug in 2.5.2/2.6/3.0: max(NaN, finite) gave -- incorrect answers when the finite number required rounding; similarly --- for the other thre functions +-- for the other three functions maxexponent: 999 minexponent: -999 precision: 6 diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 096b5381cd933..b776a2cce647d 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -114,7 +114,7 @@ def get_pooled_int(value): # These checkers return False on success, True on failure def check_rc_deltas(deltas): - # Checker for reference counters and memomry blocks. + # Checker for reference counters and memory blocks. # # bpo-30776: Try to ignore false positives: # diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 25283f8c538d8..3e2c781caa11a 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3723,7 +3723,7 @@ class MyClass: self.assertEqual(new_f, 5) self.assertEqual(some_str, 'some str') - # math.log does not have its usual reducer overriden, so the + # math.log does not have its usual reducer overridden, so the # custom reduction callback should silently direct the pickler # to the default pickling by attribute, by returning # NotImplemented @@ -3740,7 +3740,7 @@ class MyClass: def test_reducer_override_no_reference_cycle(self): # bpo-39492: reducer_override used to induce a spurious reference cycle # inside the Pickler object, that could prevent all serialized objects - # from being garbage-collected without explicity invoking gc.collect. + # from being garbage-collected without explicitly invoking gc.collect. for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): with self.subTest(proto=proto): diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index 0632577cdb303..92a64e8354acb 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -157,7 +157,7 @@ class catch_threading_exception: Context manager catching threading.Thread exception using threading.excepthook. - Attributes set when an exception is catched: + Attributes set when an exception is caught: * exc_type * exc_value diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py index 81bce2e620421..177a8a64a4329 100644 --- a/Lib/test/test__xxsubinterpreters.py +++ b/Lib/test/test__xxsubinterpreters.py @@ -1221,7 +1221,7 @@ def test_channel_list_interpreters_basic(self): import _xxsubinterpreters as _interpreters obj = _interpreters.channel_recv({cid}) """)) - # Test for channel that has boths ends associated to an interpreter. + # Test for channel that has both ends associated to an interpreter. send_interps = interpreters.channel_list_interpreters(cid, send=True) recv_interps = interpreters.channel_list_interpreters(cid, send=False) self.assertEqual(send_interps, [interp0]) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 6eaa289944218..a6ea24ceceda2 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -711,7 +711,7 @@ def test_read_all_from_pipe_reader(self): # See asyncio issue 168. This test is derived from the example # subprocess_attach_read_pipe.py, but we configure the # StreamReader's limit so that twice it is less than the size - # of the data writter. Also we must explicitly attach a child + # of the data writer. Also we must explicitly attach a child # watcher to the event loop. code = """\ diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 3cf88188ecf3b..14fa6dd76f9ca 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -228,7 +228,7 @@ def prepare_broken_pipe_test(self): # buffer large enough to feed the whole pipe buffer large_data = b'x' * support.PIPE_MAX_SIZE - # the program ends before the stdin can be feeded + # the program ends before the stdin can be fed proc = self.loop.run_until_complete( asyncio.create_subprocess_exec( sys.executable, '-c', 'pass', diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index 92b1a43500311..362fbf8df08ca 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -3269,7 +3269,7 @@ def test_run_coroutine_threadsafe_with_timeout(self): self.assertTrue(task.done()) def test_run_coroutine_threadsafe_task_cancelled(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task is cancelled.""" callback = lambda: self.target(cancel=True) future = self.loop.run_in_executor(None, callback) @@ -3277,7 +3277,7 @@ def test_run_coroutine_threadsafe_task_cancelled(self): self.loop.run_until_complete(future) def test_run_coroutine_threadsafe_task_factory_exception(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task factory raise an exception.""" def task_factory(loop, coro): diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index bdb8f768fc313..d954942286676 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -264,7 +264,7 @@ def test_return_result_with_error(self): def test_getitem_with_error(self): # Test _Py_CheckSlotResult(). Raise an exception and then calls - # PyObject_GetItem(): check that the assertion catchs the bug. + # PyObject_GetItem(): check that the assertion catches the bug. # PyObject_GetItem() must not be called with an exception set. code = textwrap.dedent(""" import _testcapi diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index b9be4c03142b6..1bfd44f954788 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -1594,7 +1594,7 @@ def assertSameSet(self, s1, s2): self.assertSetEqual(set(s1), set(s2)) def test_Set_from_iterable(self): - """Verify _from_iterable overriden to an instance method works.""" + """Verify _from_iterable overridden to an instance method works.""" class SetUsingInstanceFromIterable(MutableSet): def __init__(self, values, created_by): if not created_by: diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index 33c9fcd165621..a1d9112135af3 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -3695,7 +3695,7 @@ class B: with self.assertRaisesRegex(TypeError, msg): B(3, 4, 5) - # Explicitely make a field that follows KW_ONLY be non-keyword-only. + # Explicitly make a field that follows KW_ONLY be non-keyword-only. @dataclass class C: a: int diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 9b919e7824445..af7848c0b1b3c 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -5723,7 +5723,7 @@ class A(metaclass=M): def test_incomplete_super(self): """ - Attrubute lookup on a super object must be aware that + Attribute lookup on a super object must be aware that its target type can be uninitialized (type->tp_mro == NULL). """ class M(DebugHelperMeta): diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index a6ce6f98c8290..40143757fd36a 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1046,7 +1046,7 @@ def test_splittable_pop(self): @support.cpython_only def test_splittable_pop_pending(self): - """pop a pending key in a splitted table should not crash""" + """pop a pending key in a split table should not crash""" a, b = self.make_shared_key_dict(2) a['a'] = 4 @@ -1363,7 +1363,7 @@ def test_reversed(self): self.assertRaises(StopIteration, next, r) def test_reverse_iterator_for_empty_dict(self): - # bpo-38525: revered iterator should work properly + # bpo-38525: reversed iterator should work properly # empty dict is directly used for reference count test self.assertEqual(list(reversed({})), []) diff --git a/Lib/test/test_dict_version.py b/Lib/test/test_dict_version.py index 8cdccad0d79ab..243084c75c42b 100644 --- a/Lib/test/test_dict_version.py +++ b/Lib/test/test_dict_version.py @@ -1,5 +1,5 @@ """ -Test implementation of the PEP 509: dictionary versionning. +Test implementation of the PEP 509: dictionary versioning. """ import unittest from test.support import import_helper diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py index 1db73cc2d2220..3957077f5d612 100644 --- a/Lib/test/test_dtrace.py +++ b/Lib/test/test_dtrace.py @@ -34,7 +34,7 @@ def normalize_trace_output(output): return "\n".join(result) except (IndexError, ValueError): raise AssertionError( - "tracer produced unparseable output:\n{}".format(output) + "tracer produced unparsable output:\n{}".format(output) ) diff --git a/Lib/test/test_email/test_message.py b/Lib/test/test_email/test_message.py index 920a3d6a9cb91..4c754bf40fc30 100644 --- a/Lib/test/test_email/test_message.py +++ b/Lib/test/test_email/test_message.py @@ -433,7 +433,7 @@ class TestEmailMessageBase: --=== Content-Type: text/plain - Your message has bounced, ser. + Your message has bounced, sir. --=== Content-Type: message/rfc822 diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index aa2b3d7efbf99..c748000f7005d 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -250,7 +250,7 @@ def test_forced_io_encoding(self): def test_pre_initialization_api(self): """ - Checks some key parts of the C-API that need to work before the runtine + Checks some key parts of the C-API that need to work before the runtime is initialized (via Py_Initialize()). """ env = dict(os.environ, PYTHONPATH=os.pathsep.join(sys.path)) @@ -1170,7 +1170,7 @@ def test_init_setpath_config(self): # The current getpath.c doesn't determine the stdlib dir # in this case. 'stdlib_dir': '', - # overriden by PyConfig + # overridden by PyConfig 'program_name': 'conf_program_name', 'base_executable': 'conf_executable', 'executable': 'conf_executable', diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 70d10ebc66e98..289288478c285 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -2281,7 +2281,7 @@ def test_range_of_offsets(self): abcdefg SyntaxError: bad bad """)), - # End offset pass the source lenght + # End offset pass the source length (("bad.py", 1, 2, "abcdefg", 1, 100), dedent( """ diff --git a/Lib/test/test_future.py b/Lib/test/test_future.py index 4ef11232a3368..5a3944e69e640 100644 --- a/Lib/test/test_future.py +++ b/Lib/test/test_future.py @@ -329,7 +329,7 @@ def test_annotations(self): def test_fstring_debug_annotations(self): # f-strings with '=' don't round trip very well, so set the expected - # result explicitely. + # result explicitly. self.assertAnnotationEqual("f'{x=!r}'", expected="f'x={x!r}'") self.assertAnnotationEqual("f'{x=:}'", expected="f'x={x:}'") self.assertAnnotationEqual("f'{x=:.2f}'", expected="f'x={x:.2f}'") diff --git a/Lib/test/test_lltrace.py b/Lib/test/test_lltrace.py index 8f1a92e5c725c..06e33f4c4c2f3 100644 --- a/Lib/test/test_lltrace.py +++ b/Lib/test/test_lltrace.py @@ -12,7 +12,7 @@ def test_lltrace_does_not_crash_on_subscript_operator(self): # If this test fails, it will reproduce a crash reported as # bpo-34113. The crash happened at the command line console of # debug Python builds with __ltrace__ enabled (only possible in console), - # when the interal Python stack was negatively adjusted + # when the internal Python stack was negatively adjusted with open(os_helper.TESTFN, 'w', encoding='utf-8') as fd: self.addCleanup(os_helper.unlink, os_helper.TESTFN) fd.write(textwrap.dedent("""\ diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 2c788f2dfa65e..f844e62ca2e72 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -496,7 +496,7 @@ def test_japanese(self): class TestMiscellaneous(unittest.TestCase): def test_defaults_UTF8(self): # Issue #18378: on (at least) macOS setting LC_CTYPE to "UTF-8" is - # valid. Futhermore LC_CTYPE=UTF is used by the UTF-8 locale coercing + # valid. Furthermore LC_CTYPE=UTF is used by the UTF-8 locale coercing # during interpreter startup (on macOS). import _locale import os diff --git a/Lib/test/test_long.py b/Lib/test/test_long.py index e15bf8b418676..e5c4d7fc4220f 100644 --- a/Lib/test/test_long.py +++ b/Lib/test/test_long.py @@ -1260,7 +1260,7 @@ def equivalent_python(byte_array, byteorder, signed=False): expected) except Exception as err: raise AssertionError( - "failed to convert {} with default arugments" + "failed to convert {} with default arguments" .format(test)) from err try: diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index e716f4d385932..5f6d9f47d13d7 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -324,7 +324,7 @@ def test_match_common(self): self.assertFalse(P('b/py').match('b.py')) self.assertFalse(P('/a.py').match('b.py')) self.assertFalse(P('b.py/c').match('b.py')) - # Wilcard relative pattern. + # Wildcard relative pattern. self.assertTrue(P('b.py').match('*.py')) self.assertTrue(P('a/b.py').match('*.py')) self.assertTrue(P('/a/b.py').match('*.py')) @@ -1284,7 +1284,7 @@ def test_is_reserved(self): self.assertIs(False, P('/foo/bar').is_reserved()) # UNC paths are never reserved. self.assertIs(False, P('//my/share/nul/con/aux').is_reserved()) - # Case-insenstive DOS-device names are reserved. + # Case-insensitive DOS-device names are reserved. self.assertIs(True, P('nul').is_reserved()) self.assertIs(True, P('aux').is_reserved()) self.assertIs(True, P('prn').is_reserved()) diff --git a/Lib/test/test_strftime.py b/Lib/test/test_strftime.py index ec305e54ff24f..be43c49e40aa5 100644 --- a/Lib/test/test_strftime.py +++ b/Lib/test/test_strftime.py @@ -114,7 +114,7 @@ def strftest1(self, now): ) for e in expectations: - # musn't raise a value error + # mustn't raise a value error try: result = time.strftime(e[0], now) except ValueError as error: diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 3b80904b28d3e..cf708407c2903 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -381,7 +381,7 @@ def g456(): self.assertTrue(frame is sys._getframe()) # Verify that the captured thread frame is blocked in g456, called - # from f123. This is a litte tricky, since various bits of + # from f123. This is a little tricky, since various bits of # threading.py are also in the thread's call stack. frame = d.pop(thread_id) stack = traceback.extract_stack(frame) @@ -448,7 +448,7 @@ def g456(): self.assertEqual((None, None, None), d.pop(main_id)) # Verify that the captured thread frame is blocked in g456, called - # from f123. This is a litte tricky, since various bits of + # from f123. This is a little tricky, since various bits of # threading.py are also in the thread's call stack. exc_type, exc_value, exc_tb = d.pop(thread_id) stack = traceback.extract_stack(exc_tb.tb_frame) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index f1d483733e267..96946a281a490 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1435,7 +1435,7 @@ def test_explict_cleanup_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") temp_dir.cleanup() self.assertFalse( temp_path.exists(), @@ -1494,7 +1494,7 @@ def test_del_on_collection_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") def test_del_on_shutdown(self): # A TemporaryDirectory may be cleaned up during shutdown @@ -1559,7 +1559,7 @@ def test_del_on_shutdown_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") err = err.decode('utf-8', 'backslashreplace') self.assertNotIn("Exception", err) self.assertNotIn("Error", err) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index f7fd6510d8aa7..f98aec2651d9d 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -440,8 +440,8 @@ def test_mktime(self): @unittest.skipUnless(platform.libc_ver()[0] != 'glibc', "disabled because of a bug in glibc. Issue #13309") def test_mktime_error(self): - # It may not be possible to reliably make mktime return error - # on all platfom. This will make sure that no other exception + # It may not be possible to reliably make mktime return an error + # on all platforms. This will make sure that no other exception # than OverflowError is raised for an extreme value. tt = time.gmtime(self.t) tzname = time.strftime('%Z', tt) diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py index 63151ec89d855..d8ba487328b39 100644 --- a/Lib/test/test_unparse.py +++ b/Lib/test/test_unparse.py @@ -427,7 +427,7 @@ def test_type_ignore(self): class CosmeticTestCase(ASTTestCase): - """Test if there are cosmetic issues caused by unnecesary additions""" + """Test if there are cosmetic issues caused by unnecessary additions""" def test_simple_expressions_parens(self): self.check_src_roundtrip("(a := b)") diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py index 3bdc86d6923e0..46dac0228295d 100644 --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -1473,7 +1473,7 @@ def check_weak_del_and_len_while_iterating(self, dict, testcontext): o = Object(123456) with testcontext(): n = len(dict) - # Since underlaying dict is ordered, first item is popped + # Since underlying dict is ordered, first item is popped dict.pop(next(dict.keys())) self.assertEqual(len(dict), n - 1) dict[o] = o diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index cf40e5a5c85d2..9316d0ecbcf1a 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -556,7 +556,7 @@ def testEnviron(self): # Test handler.environ as a dict expected = {} setup_testing_defaults(expected) - # Handler inherits os_environ variables which are not overriden + # Handler inherits os_environ variables which are not overridden # by SimpleHandler.add_cgi_vars() (SimpleHandler.base_env) for key, value in os_environ.items(): if key not in expected: diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 5a8824a78ffa4..285559a872a65 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -3331,7 +3331,7 @@ class MyElement(ET.Element): self._check_element_factory_class(MyElement) def test_element_factory_pure_python_subclass(self): - # Mimick SimpleTAL's behaviour (issue #16089): both versions of + # Mimic SimpleTAL's behaviour (issue #16089): both versions of # TreeBuilder should be able to cope with a subclass of the # pure Python Element class. base = ET._Element_Py diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index 85e27ad4631d2..1f06f5fdf483e 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -561,7 +561,7 @@ def test_comparison(self): class BinaryTestCase(unittest.TestCase): - # XXX What should str(Binary(b"\xff")) return? I'm chosing "\xff" + # XXX What should str(Binary(b"\xff")) return? I'm choosing "\xff" # for now (i.e. interpreting the binary data as Latin-1-encoded # text). But this feels very unsatisfactory. Perhaps we should # only define repr(), and return r"Binary(b'\xff')" instead? diff --git a/Lib/threading.py b/Lib/threading.py index 2f473bf1b2c2b..6068d06ab6c90 100644 --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1572,7 +1572,7 @@ def _shutdown(): break for lock in locks: - # mimick Thread.join() + # mimic Thread.join() lock.acquire() lock.release() diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index 2513c972bc77f..2e7e21c6648ea 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -2735,7 +2735,7 @@ def addtag_closest(self, newtag, x, y, halo=None, start=None): """Add tag NEWTAG to item which is closest to pixel at X, Y. If several match take the top-most. All items closer than HALO are considered overlapping (all are - closests). If START is specified the next below this tag is taken.""" + closest). If START is specified the next below this tag is taken.""" self.addtag(newtag, 'closest', x, y, halo, start) def addtag_enclosed(self, newtag, x1, y1, x2, y2): @@ -3330,7 +3330,7 @@ def add_command(self, cnf={}, **kw): self.add('command', cnf or kw) def add_radiobutton(self, cnf={}, **kw): - """Addd radio menu item.""" + """Add radio menu item.""" self.add('radiobutton', cnf or kw) def add_separator(self, cnf={}, **kw): @@ -3355,7 +3355,7 @@ def insert_command(self, index, cnf={}, **kw): self.insert(index, 'command', cnf or kw) def insert_radiobutton(self, index, cnf={}, **kw): - """Addd radio menu item at INDEX.""" + """Add radio menu item at INDEX.""" self.insert(index, 'radiobutton', cnf or kw) def insert_separator(self, index, cnf={}, **kw): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index ee5af82fd1b44..082da5d0c1a00 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -968,7 +968,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets readded at its previous position + # verify that the tab gets read at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index 3e864d14d112f..d8bfaf6b67e00 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -53,7 +53,7 @@ def addAsyncCleanup(self, func, /, *args, **kwargs): # We intentionally don't add inspect.iscoroutinefunction() check # for func argument because there is no way # to check for async function reliably: - # 1. It can be "async def func()" iself + # 1. It can be "async def func()" itself # 2. Class can implement "async def __call__()" method # 3. Regular "def func()" that returns awaitable object self.addCleanup(*(func, *args), **kwargs) diff --git a/Lib/unittest/test/testmock/testsealable.py b/Lib/unittest/test/testmock/testsealable.py index 11784c3678918..daba2b49b46f6 100644 --- a/Lib/unittest/test/testmock/testsealable.py +++ b/Lib/unittest/test/testmock/testsealable.py @@ -128,7 +128,7 @@ def test_integration_with_spec_att_definition(self): m.attr_sample2 def test_integration_with_spec_method_definition(self): - """You need to defin the methods, even if they are in the spec""" + """You need to define the methods, even if they are in the spec""" m = mock.Mock(SampleObject) m.method_sample1.return_value = 1 diff --git a/Lib/venv/scripts/common/Activate.ps1 b/Lib/venv/scripts/common/Activate.ps1 index 51fc55c4ec7cb..eeea3583fa130 100644 --- a/Lib/venv/scripts/common/Activate.ps1 +++ b/Lib/venv/scripts/common/Activate.ps1 @@ -202,7 +202,7 @@ else { $Prompt = $pyvenvCfg['prompt']; } else { - Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virutal environment)" + Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)" Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'" $Prompt = Split-Path -Path $venvDir -Leaf } diff --git a/Lib/wsgiref/validate.py b/Lib/wsgiref/validate.py index 48ac0070549b3..6e16578dbb648 100644 --- a/Lib/wsgiref/validate.py +++ b/Lib/wsgiref/validate.py @@ -137,7 +137,7 @@ def validator(application): """ When applied between a WSGI server and a WSGI application, this - middleware will check for WSGI compliancy on a number of levels. + middleware will check for WSGI compliance on a number of levels. This middleware does not modify the request or response in any way, but will raise an AssertionError if anything seems off (except for a failure to close the application iterator, which diff --git a/Lib/zoneinfo/_zoneinfo.py b/Lib/zoneinfo/_zoneinfo.py index 9810637d3ef65..de68380792f17 100644 --- a/Lib/zoneinfo/_zoneinfo.py +++ b/Lib/zoneinfo/_zoneinfo.py @@ -338,7 +338,7 @@ def _utcoff_to_dstoff(trans_idx, utcoffsets, isdsts): comp_idx = trans_idx[i + 1] # If the following transition is also DST and we couldn't - # find the DST offset by this point, we're going ot have to + # find the DST offset by this point, we're going to have to # skip it and hope this transition gets assigned later if isdsts[comp_idx]: continue From webhook-mailer at python.org Wed Oct 6 19:22:20 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 06 Oct 2021 23:22:20 -0000 Subject: [Python-checkins] bpo-29505: Add fuzzer for ast.literal_eval (GH-28777) Message-ID: https://github.com/python/cpython/commit/db72e58ea5940c3942ede9f70cb897510b52fc36 commit: db72e58ea5940c3942ede9f70cb897510b52fc36 branch: main author: Ammar Askar committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-06T16:22:09-07:00 summary: bpo-29505: Add fuzzer for ast.literal_eval (GH-28777) This supercedes https://github.com/python/cpython/pull/3437 and fuzzes the method we recommend for unsafe inputs, `ast.literal_eval`. This should exercise the tokenizer and parser. files: M Modules/_xxtestfuzz/fuzz_tests.txt M Modules/_xxtestfuzz/fuzzer.c diff --git a/Modules/_xxtestfuzz/fuzz_tests.txt b/Modules/_xxtestfuzz/fuzz_tests.txt index 053b77b41b111..4e046ecf6d898 100644 --- a/Modules/_xxtestfuzz/fuzz_tests.txt +++ b/Modules/_xxtestfuzz/fuzz_tests.txt @@ -6,3 +6,4 @@ fuzz_sre_compile fuzz_sre_match fuzz_csv_reader fuzz_struct_unpack +fuzz_ast_literal_eval diff --git a/Modules/_xxtestfuzz/fuzzer.c b/Modules/_xxtestfuzz/fuzzer.c index e1256f59cc9c2..366e81a54519a 100644 --- a/Modules/_xxtestfuzz/fuzzer.c +++ b/Modules/_xxtestfuzz/fuzzer.c @@ -393,6 +393,51 @@ static int fuzz_csv_reader(const char* data, size_t size) { return 0; } +#define MAX_AST_LITERAL_EVAL_TEST_SIZE 0x10000 +PyObject* ast_literal_eval_method = NULL; +/* Called by LLVMFuzzerTestOneInput for initialization */ +static int init_ast_literal_eval(void) { + PyObject* ast_module = PyImport_ImportModule("ast"); + if (ast_module == NULL) { + return 0; + } + ast_literal_eval_method = PyObject_GetAttrString(ast_module, "literal_eval"); + return ast_literal_eval_method != NULL; +} +/* Fuzz ast.literal_eval(x) */ +static int fuzz_ast_literal_eval(const char* data, size_t size) { + if (size > MAX_AST_LITERAL_EVAL_TEST_SIZE) { + return 0; + } + /* Ignore non null-terminated strings since ast can't handle + embedded nulls */ + if (memchr(data, '\0', size) == NULL) { + return 0; + } + + PyObject* s = PyUnicode_FromString(data); + /* Ignore exceptions until we have a valid string */ + if (s == NULL) { + PyErr_Clear(); + return 0; + } + + PyObject* literal = PyObject_CallOneArg(ast_literal_eval_method, s); + /* Ignore some common errors thrown by ast.literal_eval */ + if (literal == NULL && (PyErr_ExceptionMatches(PyExc_ValueError) || + PyErr_ExceptionMatches(PyExc_TypeError) || + PyErr_ExceptionMatches(PyExc_SyntaxError) || + PyErr_ExceptionMatches(PyExc_MemoryError) || + PyErr_ExceptionMatches(PyExc_RecursionError)) + ) { + PyErr_Clear(); + } + + Py_XDECREF(literal); + Py_DECREF(s); + return 0; +} + /* Run fuzzer and abort on failure. */ static int _run_fuzz(const uint8_t *data, size_t size, int(*fuzzer)(const char* , size_t)) { int rv = fuzzer((const char*) data, size); @@ -507,6 +552,17 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) { } rv |= _run_fuzz(data, size, fuzz_csv_reader); +#endif +#if !defined(_Py_FUZZ_ONE) || defined(_Py_FUZZ_fuzz_ast_literal_eval) + static int AST_LITERAL_EVAL_INITIALIZED = 0; + if (!AST_LITERAL_EVAL_INITIALIZED && !init_ast_literal_eval()) { + PyErr_Print(); + abort(); + } else { + AST_LITERAL_EVAL_INITIALIZED = 1; + } + + rv |= _run_fuzz(data, size, fuzz_ast_literal_eval); #endif return rv; } From webhook-mailer at python.org Wed Oct 6 19:57:19 2021 From: webhook-mailer at python.org (gpshead) Date: Wed, 06 Oct 2021 23:57:19 -0000 Subject: [Python-checkins] Fix typos in the Objects directory (GH-28766) Message-ID: https://github.com/python/cpython/commit/5f401f10400123afa9171548c432ea3fc37c0736 commit: 5f401f10400123afa9171548c432ea3fc37c0736 branch: main author: Christian Clauss committer: gpshead date: 2021-10-06T16:57:10-07:00 summary: Fix typos in the Objects directory (GH-28766) files: M Objects/exception_handling_notes.txt M Objects/exceptions.c M Objects/floatobject.c M Objects/frameobject.c M Objects/listobject.c M Objects/listsort.txt M Objects/obmalloc.c M Objects/setobject.c M Objects/unicodeobject.c M PC/getpathp.c M Parser/tokenizer.h diff --git a/Objects/exception_handling_notes.txt b/Objects/exception_handling_notes.txt index 2183fa111f105..e738c2781ad23 100644 --- a/Objects/exception_handling_notes.txt +++ b/Objects/exception_handling_notes.txt @@ -105,7 +105,7 @@ All offsets and lengths are in instructions, not bytes. We want the format to be compact, but quickly searchable. For it to be compact, it needs to have variable sized entries so that we can store common (small) offsets compactly, but handle large offsets if needed. For it to be searchable quickly, we need to support binary search giving us log(n) performance in all cases. -Binary search typically assumes fixed size entries, but that is not necesary, as long as we can identify the start of an entry. +Binary search typically assumes fixed size entries, but that is not necessary, as long as we can identify the start of an entry. It is worth noting that the size (end-start) is always smaller than the end, so we encode the entries as: start, size, target, depth, push-lasti diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 714039ee1abca..a9ea42c98422d 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -90,7 +90,7 @@ static void BaseException_dealloc(PyBaseExceptionObject *self) { PyObject_GC_UnTrack(self); - // bpo-44348: The trashcan mecanism prevents stack overflow when deleting + // bpo-44348: The trashcan mechanism prevents stack overflow when deleting // long chains of exceptions. For example, exceptions can be chained // through the __context__ attributes or the __traceback__ attribute. Py_TRASHCAN_BEGIN(self, BaseException_dealloc) diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 92faa7c132023..e4ce7e74d2c57 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -2350,7 +2350,7 @@ _PyFloat_Pack8(double x, unsigned char *p, int le) flo = 0; ++fhi; if (fhi >> 28) { - /* And it also progagated out of the next 28 bits. */ + /* And it also propagated out of the next 28 bits. */ fhi = 0; ++e; if (e >= 2047) diff --git a/Objects/frameobject.c b/Objects/frameobject.c index e4c16de66211d..5271790f018af 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -408,7 +408,7 @@ frame_stack_pop(PyFrameObject *f) * would still work without any stack errors), but there are some constructs * that limit jumping: * - * o Any excpetion handlers. + * o Any exception handlers. * o 'for' and 'async for' loops can't be jumped into because the * iterator needs to be on the stack. * o Jumps cannot be made from within a trace function invoked with a diff --git a/Objects/listobject.c b/Objects/listobject.c index e7c4742f1db45..ed5324155f627 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -1973,7 +1973,7 @@ powerloop(Py_ssize_t s1, Py_ssize_t n1, Py_ssize_t n2, Py_ssize_t n) * and merge adjacent runs on the stack with greater power. See listsort.txt * for more info. * - * It's the caller's responsibilty to push the new run on the stack when this + * It's the caller's responsibility to push the new run on the stack when this * returns. * * Returns 0 on success, -1 on error. @@ -2067,7 +2067,7 @@ safe_object_compare(PyObject *v, PyObject *w, MergeState *ms) return PyObject_RichCompareBool(v, w, Py_LT); } -/* Homogeneous compare: safe for any two compareable objects of the same type. +/* Homogeneous compare: safe for any two comparable objects of the same type. * (ms->key_richcompare is set to ob_type->tp_richcompare in the * pre-sort check.) */ diff --git a/Objects/listsort.txt b/Objects/listsort.txt index 306e5e44d2ecf..32a59e510f075 100644 --- a/Objects/listsort.txt +++ b/Objects/listsort.txt @@ -382,7 +382,7 @@ things we don't have: extension on most platforms, but not all, and there's no uniform spelling on the platforms that support it. -- Integer divison on an integer type twice as wide as needed to hold the +- Integer division on an integer type twice as wide as needed to hold the list length. But the latter is Py_ssize_t for us, and is typically the widest native signed integer type the platform supports. @@ -797,6 +797,6 @@ OPTIMIZATION OF INDIVIDUAL COMPARISONS As noted above, even the simplest Python comparison triggers a large pile of C-level pointer dereferences, conditionals, and function calls. This can be partially mitigated by pre-scanning the data to determine whether the data is -homogenous with respect to type. If so, it is sometimes possible to +homogeneous with respect to type. If so, it is sometimes possible to substitute faster type-specific comparisons for the slower, generic PyObject_RichCompareBool. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 903ca1c9e4b98..2d6fedd6cf0a1 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -848,7 +848,7 @@ static int running_on_valgrind = -1; /* * Alignment of addresses returned to the user. 8-bytes alignment works - * on most current architectures (with 32-bit or 64-bit address busses). + * on most current architectures (with 32-bit or 64-bit address buses). * The alignment value is also used for grouping small requests in size * classes spaced ALIGNMENT bytes apart. * diff --git a/Objects/setobject.c b/Objects/setobject.c index f71417d9f6ded..0be067857d694 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1409,7 +1409,7 @@ set_difference_update_internal(PySetObject *so, PyObject *other) /* Optimization: When the other set is more than 8 times larger than the base set, replace the other set with - interesection of the two sets. + intersection of the two sets. */ if ((PySet_GET_SIZE(other) >> 3) > PySet_GET_SIZE(so)) { other = set_intersection(so, other); diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 02bf56e681e56..741cf9dd13d9f 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7349,7 +7349,7 @@ PyUnicode_AsASCIIString(PyObject *unicode) #endif /* INT_MAX is the theoretical largest chunk (or INT_MAX / 2 when - transcoding from UTF-16), but INT_MAX / 4 perfoms better in + transcoding from UTF-16), but INT_MAX / 4 performs better in both cases also and avoids partial characters overrunning the length limit in MultiByteToWideChar on Windows */ #define DECODING_CHUNK_SIZE (INT_MAX/4) @@ -15876,7 +15876,7 @@ init_fs_codec(PyInterpreterState *interp) _Py_error_handler error_handler; error_handler = get_error_handler_wide(config->filesystem_errors); if (error_handler == _Py_ERROR_UNKNOWN) { - PyErr_SetString(PyExc_RuntimeError, "unknow filesystem error handler"); + PyErr_SetString(PyExc_RuntimeError, "unknown filesystem error handler"); return -1; } diff --git a/PC/getpathp.c b/PC/getpathp.c index 062697b3e9afd..79569bb554457 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -354,7 +354,7 @@ extern const char *PyWin_DLLVersionString; Returns NULL, or a pointer that should be freed. XXX - this code is pretty strange, as it used to also - work on Win16, where the buffer sizes werent available + work on Win16, where the buffer sizes were not available in advance. It could be simplied now Win16/Win32s is dead! */ static wchar_t * diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h index a40f7d9687b44..677f9dba490be 100644 --- a/Parser/tokenizer.h +++ b/Parser/tokenizer.h @@ -23,7 +23,7 @@ enum interactive_underflow_t { /* Normal mode of operation: return a new token when asked in interactie mode */ IUNDERFLOW_NORMAL, /* Forcefully return ENDMARKER when asked for a new token in interactive mode. This - * can be used to prevent the tokenizer to promt the user for new tokens */ + * can be used to prevent the tokenizer to prompt the user for new tokens */ IUNDERFLOW_STOP, }; From webhook-mailer at python.org Wed Oct 6 20:52:05 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 07 Oct 2021 00:52:05 -0000 Subject: [Python-checkins] bpo-45385: Fix reference leak from descr_check (#28719) Message-ID: https://github.com/python/cpython/commit/e6ff4eba6da9b64aed235ba8d730b5645f71955c commit: e6ff4eba6da9b64aed235ba8d730b5645f71955c branch: main author: Dong-hee Na committer: corona10 date: 2021-10-07T09:51:56+09:00 summary: bpo-45385: Fix reference leak from descr_check (#28719) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst M Objects/descrobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst new file mode 100644 index 0000000000000..8047c102634f0 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst @@ -0,0 +1 @@ +Fix reference leak from descr_check. Patch by Dong-hee Na. diff --git a/Objects/descrobject.c b/Objects/descrobject.c index 4ed36e94b3f79..946ea6aa80319 100644 --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -72,13 +72,8 @@ wrapperdescr_repr(PyWrapperDescrObject *descr) } static int -descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) +descr_check(PyDescrObject *descr, PyObject *obj) { - if (obj == NULL) { - Py_INCREF(descr); - *pres = (PyObject *)descr; - return 1; - } if (!PyObject_TypeCheck(obj, descr->d_type)) { PyErr_Format(PyExc_TypeError, "descriptor '%V' for '%.100s' objects " @@ -86,8 +81,7 @@ descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) descr_name((PyDescrObject *)descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = NULL; - return 1; + return -1; } return 0; } @@ -137,10 +131,12 @@ classmethod_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_method->ml_flags & METH_METHOD) { if (PyType_Check(type)) { return PyCMethod_New(descr->d_method, obj, NULL, descr->d_common.d_type); @@ -159,10 +155,12 @@ method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_member->flags & PY_AUDIT_READ) { if (PySys_Audit("object.__getattr__", "Os", @@ -177,10 +175,12 @@ member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_getset->get != NULL) return descr->d_getset->get(obj, descr->d_getset->closure); PyErr_Format(PyExc_AttributeError, @@ -193,16 +193,17 @@ getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * wrapperdescr_get(PyWrapperDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } return PyWrapper_New((PyObject *)descr, obj); } static int -descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, - int *pres) +descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value) { assert(obj != NULL); if (!PyObject_TypeCheck(obj, descr->d_type)) { @@ -212,8 +213,7 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, descr_name(descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = -1; - return 1; + return -1; } return 0; } @@ -221,23 +221,22 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, static int member_set(PyMemberDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } return PyMember_SetOne((char *)obj, descr->d_member, value); } static int getset_set(PyGetSetDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; - if (descr->d_getset->set != NULL) + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } + if (descr->d_getset->set != NULL) { return descr->d_getset->set(obj, value, descr->d_getset->closure); + } PyErr_Format(PyExc_AttributeError, "attribute '%V' of '%.100s' objects is not writable", descr_name((PyDescrObject *)descr), "?", @@ -264,8 +263,7 @@ method_check_args(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObj return -1; } PyObject *self = args[0]; - PyObject *dummy; - if (descr_check((PyDescrObject *)func, self, &dummy)) { + if (descr_check((PyDescrObject *)func, self) < 0) { return -1; } if (kwnames && PyTuple_GET_SIZE(kwnames)) { From webhook-mailer at python.org Wed Oct 6 21:38:52 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 01:38:52 -0000 Subject: [Python-checkins] bpo-35970: Add help flag to base64 module (GH-28774) Message-ID: https://github.com/python/cpython/commit/5baec4aea6821256f5d1785a6bd596fab069f1b6 commit: 5baec4aea6821256f5d1785a6bd596fab069f1b6 branch: main author: Ammar Askar committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-06T18:38:43-07:00 summary: bpo-35970: Add help flag to base64 module (GH-28774) This continues off rkuska's work from https://github.com/python/cpython/pull/11789 seeing as the patch wasn't updated to add tests. files: A Misc/NEWS.d/next/Library/2019-02-11-19-06-10.bpo-35970.ZRvh51.rst M Lib/base64.py M Lib/test/test_base64.py diff --git a/Lib/base64.py b/Lib/base64.py index b25156ddb35d3..7e9c2a2ca477f 100755 --- a/Lib/base64.py +++ b/Lib/base64.py @@ -567,15 +567,17 @@ def decodebytes(s): def main(): """Small main program""" import sys, getopt + usage = """usage: %s [-h|-d|-e|-u|-t] [file|-] + -h: print this help message and exit + -d, -u: decode + -e: encode (default) + -t: encode and decode string 'Aladdin:open sesame'"""%sys.argv[0] try: - opts, args = getopt.getopt(sys.argv[1:], 'deut') + opts, args = getopt.getopt(sys.argv[1:], 'hdeut') except getopt.error as msg: sys.stdout = sys.stderr print(msg) - print("""usage: %s [-d|-e|-u|-t] [file|-] - -d, -u: decode - -e: encode (default) - -t: encode and decode string 'Aladdin:open sesame'"""%sys.argv[0]) + print(usage) sys.exit(2) func = encode for o, a in opts: @@ -583,6 +585,7 @@ def main(): if o == '-d': func = decode if o == '-u': func = decode if o == '-t': test(); return + if o == '-h': print(usage); return if args and args[0] != '-': with open(args[0], 'rb') as f: func(f, sys.stdout.buffer) diff --git a/Lib/test/test_base64.py b/Lib/test/test_base64.py index 418492432a167..217f294546884 100644 --- a/Lib/test/test_base64.py +++ b/Lib/test/test_base64.py @@ -788,5 +788,15 @@ def test_decode(self): output = self.get_output('-d', os_helper.TESTFN) self.assertEqual(output.rstrip(), b'a\xffb') + def test_prints_usage_with_help_flag(self): + output = self.get_output('-h') + self.assertIn(b'usage: ', output) + self.assertIn(b'-d, -u: decode', output) + + def test_prints_usage_with_invalid_flag(self): + output = script_helper.assert_python_failure('-m', 'base64', '-x').err + self.assertIn(b'usage: ', output) + self.assertIn(b'-d, -u: decode', output) + if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Library/2019-02-11-19-06-10.bpo-35970.ZRvh51.rst b/Misc/NEWS.d/next/Library/2019-02-11-19-06-10.bpo-35970.ZRvh51.rst new file mode 100644 index 0000000000000..bdae153a500f4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-02-11-19-06-10.bpo-35970.ZRvh51.rst @@ -0,0 +1,2 @@ +Add help flag to the base64 module's command line interface. Patch contributed +by Robert Kuska. From webhook-mailer at python.org Wed Oct 6 22:10:23 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 07 Oct 2021 02:10:23 -0000 Subject: [Python-checkins] [3.9] bpo-45385: Fix reference leak from descr_check (GH-28719) (GH-28780) Message-ID: https://github.com/python/cpython/commit/660718dba57624720c2a9832f10f29ace59c77cf commit: 660718dba57624720c2a9832f10f29ace59c77cf branch: 3.9 author: Dong-hee Na committer: corona10 date: 2021-10-07T11:10:15+09:00 summary: [3.9] bpo-45385: Fix reference leak from descr_check (GH-28719) (GH-28780) (cherry picked from commit e6ff4eba6da9b64aed235ba8d730b5645f71955c) Co-authored-by: Dong-hee Na files: A Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst M Objects/descrobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst new file mode 100644 index 0000000000000..8047c102634f0 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst @@ -0,0 +1 @@ +Fix reference leak from descr_check. Patch by Dong-hee Na. diff --git a/Objects/descrobject.c b/Objects/descrobject.c index fce9cdd309077..075a92d4084d5 100644 --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -72,13 +72,8 @@ wrapperdescr_repr(PyWrapperDescrObject *descr) } static int -descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) +descr_check(PyDescrObject *descr, PyObject *obj) { - if (obj == NULL) { - Py_INCREF(descr); - *pres = (PyObject *)descr; - return 1; - } if (!PyObject_TypeCheck(obj, descr->d_type)) { PyErr_Format(PyExc_TypeError, "descriptor '%V' for '%.100s' objects " @@ -86,8 +81,7 @@ descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) descr_name((PyDescrObject *)descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = NULL; - return 1; + return -1; } return 0; } @@ -137,10 +131,13 @@ classmethod_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + Py_INCREF(descr); + return (PyObject *)descr; + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_method->ml_flags & METH_METHOD) { if (PyType_Check(type)) { return PyCMethod_New(descr->d_method, obj, NULL, descr->d_common.d_type); @@ -159,10 +156,13 @@ method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + Py_INCREF(descr); + return (PyObject *)descr; + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_member->flags & READ_RESTRICTED) { if (PySys_Audit("object.__getattr__", "Os", @@ -177,10 +177,13 @@ member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + Py_INCREF(descr); + return (PyObject *)descr; + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_getset->get != NULL) return descr->d_getset->get(obj, descr->d_getset->closure); PyErr_Format(PyExc_AttributeError, @@ -193,16 +196,18 @@ getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * wrapperdescr_get(PyWrapperDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + Py_INCREF(descr); + return (PyObject *)descr; + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } return PyWrapper_New((PyObject *)descr, obj); } static int -descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, - int *pres) +descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value) { assert(obj != NULL); if (!PyObject_TypeCheck(obj, descr->d_type)) { @@ -212,8 +217,7 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, descr_name(descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = -1; - return 1; + return -1; } return 0; } @@ -221,23 +225,22 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, static int member_set(PyMemberDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } return PyMember_SetOne((char *)obj, descr->d_member, value); } static int getset_set(PyGetSetDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; - if (descr->d_getset->set != NULL) + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } + if (descr->d_getset->set != NULL) { return descr->d_getset->set(obj, value, descr->d_getset->closure); + } PyErr_Format(PyExc_AttributeError, "attribute '%V' of '%.100s' objects is not writable", descr_name((PyDescrObject *)descr), "?", @@ -264,8 +267,7 @@ method_check_args(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObj return -1; } PyObject *self = args[0]; - PyObject *dummy; - if (descr_check((PyDescrObject *)func, self, &dummy)) { + if (descr_check((PyDescrObject *)func, self) < 0) { return -1; } if (kwnames && PyTuple_GET_SIZE(kwnames)) { From webhook-mailer at python.org Thu Oct 7 03:48:17 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 07 Oct 2021 07:48:17 -0000 Subject: [Python-checkins] bpo-45385: Fix reference leak from descr_check (GH-28719) (GH-28779) Message-ID: https://github.com/python/cpython/commit/35d4857375b6ef8f1243db4da9c2cba0bee63ad6 commit: 35d4857375b6ef8f1243db4da9c2cba0bee63ad6 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: corona10 date: 2021-10-07T16:48:00+09:00 summary: bpo-45385: Fix reference leak from descr_check (GH-28719) (GH-28779) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst M Objects/descrobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst new file mode 100644 index 0000000000000..8047c102634f0 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-06-21-20-11.bpo-45385.CTUT8s.rst @@ -0,0 +1 @@ +Fix reference leak from descr_check. Patch by Dong-hee Na. diff --git a/Objects/descrobject.c b/Objects/descrobject.c index 57a9607d10c31..97669bef368da 100644 --- a/Objects/descrobject.c +++ b/Objects/descrobject.c @@ -72,13 +72,8 @@ wrapperdescr_repr(PyWrapperDescrObject *descr) } static int -descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) +descr_check(PyDescrObject *descr, PyObject *obj) { - if (obj == NULL) { - Py_INCREF(descr); - *pres = (PyObject *)descr; - return 1; - } if (!PyObject_TypeCheck(obj, descr->d_type)) { PyErr_Format(PyExc_TypeError, "descriptor '%V' for '%.100s' objects " @@ -86,8 +81,7 @@ descr_check(PyDescrObject *descr, PyObject *obj, PyObject **pres) descr_name((PyDescrObject *)descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = NULL; - return 1; + return -1; } return 0; } @@ -137,10 +131,12 @@ classmethod_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_method->ml_flags & METH_METHOD) { if (PyType_Check(type)) { return PyCMethod_New(descr->d_method, obj, NULL, descr->d_common.d_type); @@ -159,10 +155,12 @@ method_get(PyMethodDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_member->flags & PY_AUDIT_READ) { if (PySys_Audit("object.__getattr__", "Os", @@ -177,10 +175,12 @@ member_get(PyMemberDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } if (descr->d_getset->get != NULL) return descr->d_getset->get(obj, descr->d_getset->closure); PyErr_Format(PyExc_AttributeError, @@ -193,16 +193,17 @@ getset_get(PyGetSetDescrObject *descr, PyObject *obj, PyObject *type) static PyObject * wrapperdescr_get(PyWrapperDescrObject *descr, PyObject *obj, PyObject *type) { - PyObject *res; - - if (descr_check((PyDescrObject *)descr, obj, &res)) - return res; + if (obj == NULL) { + return Py_NewRef(descr); + } + if (descr_check((PyDescrObject *)descr, obj) < 0) { + return NULL; + } return PyWrapper_New((PyObject *)descr, obj); } static int -descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, - int *pres) +descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value) { assert(obj != NULL); if (!PyObject_TypeCheck(obj, descr->d_type)) { @@ -212,8 +213,7 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, descr_name(descr), "?", descr->d_type->tp_name, Py_TYPE(obj)->tp_name); - *pres = -1; - return 1; + return -1; } return 0; } @@ -221,23 +221,22 @@ descr_setcheck(PyDescrObject *descr, PyObject *obj, PyObject *value, static int member_set(PyMemberDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } return PyMember_SetOne((char *)obj, descr->d_member, value); } static int getset_set(PyGetSetDescrObject *descr, PyObject *obj, PyObject *value) { - int res; - - if (descr_setcheck((PyDescrObject *)descr, obj, value, &res)) - return res; - if (descr->d_getset->set != NULL) + if (descr_setcheck((PyDescrObject *)descr, obj, value) < 0) { + return -1; + } + if (descr->d_getset->set != NULL) { return descr->d_getset->set(obj, value, descr->d_getset->closure); + } PyErr_Format(PyExc_AttributeError, "attribute '%V' of '%.100s' objects is not writable", descr_name((PyDescrObject *)descr), "?", @@ -264,8 +263,7 @@ method_check_args(PyObject *func, PyObject *const *args, Py_ssize_t nargs, PyObj return -1; } PyObject *self = args[0]; - PyObject *dummy; - if (descr_check((PyDescrObject *)func, self, &dummy)) { + if (descr_check((PyDescrObject *)func, self) < 0) { return -1; } if (kwnames && PyTuple_GET_SIZE(kwnames)) { From webhook-mailer at python.org Thu Oct 7 04:35:04 2021 From: webhook-mailer at python.org (gpshead) Date: Thu, 07 Oct 2021 08:35:04 -0000 Subject: [Python-checkins] Fix typos in the Modules directory (GH-28761) Message-ID: https://github.com/python/cpython/commit/dd02a696e55b450413e765e698e653d781ca4205 commit: dd02a696e55b450413e765e698e653d781ca4205 branch: main author: Christian Clauss committer: gpshead date: 2021-10-07T01:34:42-07:00 summary: Fix typos in the Modules directory (GH-28761) files: M Modules/_csv.c M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_ctypes/ctypes.h M Modules/_pickle.c M Modules/_sre.c M Modules/_testcapimodule.c M Modules/_threadmodule.c M Modules/_tkinter.c M Modules/_tracemalloc.c M Modules/_zoneinfo.c M Modules/expat/xmlparse.c M Modules/gc_weakref.txt M Modules/getpath.c M Modules/sha1module.c M Modules/signalmodule.c M Modules/socketmodule.c M Modules/syslogmodule.c M Modules/zlibmodule.c diff --git a/Modules/_csv.c b/Modules/_csv.c index 3109fd16bc744..3729d2be36252 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -489,7 +489,7 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) } /* Since dialect is now a heap type, it inherits pickling method for - * protocol 0 and 1 from object, therefore it needs to be overriden */ + * protocol 0 and 1 from object, therefore it needs to be overridden */ PyDoc_STRVAR(dialect_reduce_doc, "raises an exception to avoid pickling"); diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index 17e82f90cf474..e2204961070db 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -834,7 +834,7 @@ static int _call_function_pointer(int flags, # define HAVE_FFI_PREP_CIF_VAR_RUNTIME false # endif - /* Even on Apple-arm64 the calling convention for variadic functions conincides + /* Even on Apple-arm64 the calling convention for variadic functions coincides * with the standard calling convention in the case that the function called * only with its fixed arguments. Thus, we do not need a special flag to be * set on variadic functions. We treat a function as variadic if it is called diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index a21a9daaed02c..ec6feca8b0f80 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -35,7 +35,7 @@ PyCField_new(PyTypeObject *type, PyObject *args, PyObject *kwds) * Expects the size, index and offset for the current field in *psize and * *poffset, stores the total size so far in *psize, the offset for the next * field in *poffset, the alignment requirements for the current field in - * *palign, and returns a field desriptor for this field. + * *palign, and returns a field descriptor for this field. */ /* * bitfields extension: diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 6110027980827..9600ddc7413b2 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -208,7 +208,7 @@ typedef struct { PyObject *checker; int flags; /* calling convention and such */ - /* pep3118 fields, pointers neeed PyMem_Free */ + /* pep3118 fields, pointers need PyMem_Free */ char *format; int ndim; Py_ssize_t *shape; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 3e74fafb38417..e693b506a1e91 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4527,7 +4527,7 @@ dump(PicklerObject *self, PyObject *obj) * call when setting the reducer_override attribute of the Pickler instance * to a bound method of the same instance. This is important as the Pickler * instance holds a reference to each object it has pickled (through its - * memo): thus, these objects wont be garbage-collected as long as the + * memo): thus, these objects won't be garbage-collected as long as the * Pickler itself is not collected. */ Py_CLEAR(self->reducer_override); return status; @@ -6540,7 +6540,7 @@ do_setitems(UnpicklerObject *self, Py_ssize_t x) return 0; if ((len - x) % 2 != 0) { PickleState *st = _Pickle_GetGlobalState(); - /* Currupt or hostile pickle -- we never write one like this. */ + /* Corrupt or hostile pickle -- we never write one like this. */ PyErr_SetString(st->UnpicklingError, "odd number of items for SETITEMS"); return -1; diff --git a/Modules/_sre.c b/Modules/_sre.c index f4ec862dcabf0..213730860cfb5 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -15,7 +15,7 @@ * 2001-05-14 fl fixes for 1.5.2 compatibility * 2001-07-01 fl added BIGCHARSET support (from Martin von Loewis) * 2001-10-18 fl fixed group reset issue (from Matthew Mueller) - * 2001-10-20 fl added split primitive; reenable unicode for 1.6/2.0/2.1 + * 2001-10-20 fl added split primitive; re-enable unicode for 1.6/2.0/2.1 * 2001-10-21 fl added sub/subn primitive * 2001-10-24 fl added finditer primitive (for 2.2 only) * 2001-12-07 fl fixed memory leak in sub/subn (Guido van Rossum) diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 03dc7763bfd05..4a2907f3f06bd 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5323,7 +5323,7 @@ encode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; @@ -5366,7 +5366,7 @@ decode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 543d82d0b9382..ff858386b426f 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -1219,7 +1219,7 @@ where the corresponding signal handler will be executed.\n\ If *signum* is omitted, SIGINT is assumed.\n\ A subthread can use this function to interrupt the main thread.\n\ \n\ -Note: the default signal hander for SIGINT raises ``KeyboardInterrupt``." +Note: the default signal handler for SIGINT raises ``KeyboardInterrupt``." ); static lockobject *newlockobject(PyObject *module); diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 7be9b8c0385b9..aabf20b8d963c 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -797,7 +797,7 @@ PyTclObject_str(PyTclObject *self) Py_INCREF(self->string); return self->string; } - /* XXX Could chache result if it is non-ASCII. */ + /* XXX Could cache result if it is non-ASCII. */ return unicodeFromTclObj(self->value); } diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 09d18fb8f278f..9ba0ebbbaab5c 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -833,7 +833,7 @@ tracemalloc_clear_filename(void *value) static void tracemalloc_clear_traces(void) { - /* The GIL protects variables againt concurrent access */ + /* The GIL protects variables against concurrent access */ assert(PyGILState_Check()); TABLES_LOCK(); diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index c722330ee491a..04fa09422b213 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -1342,7 +1342,7 @@ tzrule_transitions(_tzrule *rule, int year, int64_t *start, int64_t *end) * could technically be calculated from the timestamp, but given that the * callers of this function already have the year information accessible from * the datetime struct, it is taken as an additional parameter to reduce - * unncessary calculation. + * unnecessary calculation. * */ static _ttinfo * find_tzrule_ttinfo(_tzrule *rule, int64_t ts, unsigned char fold, int year) diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 5ba56eaea6357..034a03c30851a 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -3976,7 +3976,7 @@ initializeEncoding(XML_Parser parser) { const char *s; #ifdef XML_UNICODE char encodingBuf[128]; - /* See comments abount `protoclEncodingName` in parserInit() */ + /* See comments about `protoclEncodingName` in parserInit() */ if (! parser->m_protocolEncodingName) s = NULL; else { diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt index b5b9f7b2e3d4d..6d07cce123643 100644 --- a/Modules/gc_weakref.txt +++ b/Modules/gc_weakref.txt @@ -50,7 +50,7 @@ CT while gc is running. https://www.python.org/sf/1055820 shows how innocent it can be, and also how nasty. Variants of the three -focussed test cases attached to that bug report are now part of Python's +focused test cases attached to that bug report are now part of Python's standard Lib/test/test_gc.py. Jim Fulton gave the best nutshell summary of the new (in 2.4 and 2.3.5) diff --git a/Modules/getpath.c b/Modules/getpath.c index 56775e9cb44af..22e5ef24b5d57 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -1457,7 +1457,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig) } /* If a pyvenv.cfg configure file is found, - argv0_path is overriden with its 'home' variable. */ + argv0_path is overridden with its 'home' variable. */ status = calculate_read_pyenv(calculate); if (_PyStatus_EXCEPTION(status)) { return status; diff --git a/Modules/sha1module.c b/Modules/sha1module.c index 6980051dc08f8..d186aa460a07b 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -269,7 +269,7 @@ sha1_done(struct sha1_state *sha1, unsigned char *out) sha1->curlen = 0; } - /* pad upto 56 bytes of zeroes */ + /* pad up to 56 bytes of zeroes */ while (sha1->curlen < 56) { sha1->buf[sha1->curlen++] = (unsigned char)0; } diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 8b7ef2cc688fe..d5e6a43af2865 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1595,7 +1595,7 @@ signal_get_set_handlers(signal_state_t *state, PyObject *mod_dict) Py_XDECREF(old_func); } - // Instal Python SIGINT handler which raises KeyboardInterrupt + // Install Python SIGINT handler which raises KeyboardInterrupt PyObject* sigint_func = get_handler(SIGINT); if (sigint_func == state->default_handler) { PyObject *int_handler = PyMapping_GetItemString(mod_dict, diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 98212274b46a9..50962c41cd3a6 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -929,7 +929,7 @@ sock_call_ex(PySocketSockObject *s, reading, but the data then discarded by the OS because of a wrong checksum. - Loop on select() to recheck for socket readyness. */ + Loop on select() to recheck for socket readiness. */ continue; } diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c index cdc94a60a373d..0f2fea15b3a6f 100644 --- a/Modules/syslogmodule.c +++ b/Modules/syslogmodule.c @@ -183,7 +183,7 @@ syslog_syslog(PyObject * self, PyObject * args) PyObject *openargs; /* Continue even if PyTuple_New fails, because openlog(3) is optional. - * So, we can still do loggin in the unlikely event things are so hosed + * So, we can still do logging in the unlikely event things are so hosed * that we can't do this tuple. */ if ((openargs = PyTuple_New(0))) { diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c index 28fe8840fc067..67bde701fa608 100644 --- a/Modules/zlibmodule.c +++ b/Modules/zlibmodule.c @@ -82,7 +82,7 @@ typedef struct { Bytef *next_posi; } _Uint32Window; -/* Initialize the buffer with an inital buffer size. +/* Initialize the buffer with an initial buffer size. On success, return value >= 0 On failure, return value < 0 */ From webhook-mailer at python.org Thu Oct 7 04:55:23 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 08:55:23 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Modules directory (GH-28761) (GH-28781) Message-ID: https://github.com/python/cpython/commit/5afc5bb45d614ea4a512e257b6b60e4f98185043 commit: 5afc5bb45d614ea4a512e257b6b60e4f98185043 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T01:55:18-07:00 summary: [3.10] Fix typos in the Modules directory (GH-28761) (GH-28781) (cherry picked from commit dd02a696e55b450413e765e698e653d781ca4205) Co-authored-by: Christian Clauss Automerge-Triggered-By: GH:gpshead files: M Modules/_csv.c M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_ctypes/ctypes.h M Modules/_pickle.c M Modules/_sre.c M Modules/_testcapimodule.c M Modules/_threadmodule.c M Modules/_tkinter.c M Modules/_tracemalloc.c M Modules/_zoneinfo.c M Modules/expat/xmlparse.c M Modules/gc_weakref.txt M Modules/getpath.c M Modules/sha1module.c M Modules/signalmodule.c M Modules/socketmodule.c M Modules/syslogmodule.c M Modules/zlibmodule.c diff --git a/Modules/_csv.c b/Modules/_csv.c index 3109fd16bc744..3729d2be36252 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -489,7 +489,7 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) } /* Since dialect is now a heap type, it inherits pickling method for - * protocol 0 and 1 from object, therefore it needs to be overriden */ + * protocol 0 and 1 from object, therefore it needs to be overridden */ PyDoc_STRVAR(dialect_reduce_doc, "raises an exception to avoid pickling"); diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index f8f8efa4ee879..ddf289e3e8f60 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -834,7 +834,7 @@ static int _call_function_pointer(int flags, # define HAVE_FFI_PREP_CIF_VAR_RUNTIME false # endif - /* Even on Apple-arm64 the calling convention for variadic functions conincides + /* Even on Apple-arm64 the calling convention for variadic functions coincides * with the standard calling convention in the case that the function called * only with its fixed arguments. Thus, we do not need a special flag to be * set on variadic functions. We treat a function as variadic if it is called diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index a21a9daaed02c..ec6feca8b0f80 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -35,7 +35,7 @@ PyCField_new(PyTypeObject *type, PyObject *args, PyObject *kwds) * Expects the size, index and offset for the current field in *psize and * *poffset, stores the total size so far in *psize, the offset for the next * field in *poffset, the alignment requirements for the current field in - * *palign, and returns a field desriptor for this field. + * *palign, and returns a field descriptor for this field. */ /* * bitfields extension: diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 6110027980827..9600ddc7413b2 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -208,7 +208,7 @@ typedef struct { PyObject *checker; int flags; /* calling convention and such */ - /* pep3118 fields, pointers neeed PyMem_Free */ + /* pep3118 fields, pointers need PyMem_Free */ char *format; int ndim; Py_ssize_t *shape; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 691d4a293e849..919490c88f7d0 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4527,7 +4527,7 @@ dump(PicklerObject *self, PyObject *obj) * call when setting the reducer_override attribute of the Pickler instance * to a bound method of the same instance. This is important as the Pickler * instance holds a reference to each object it has pickled (through its - * memo): thus, these objects wont be garbage-collected as long as the + * memo): thus, these objects won't be garbage-collected as long as the * Pickler itself is not collected. */ Py_CLEAR(self->reducer_override); return status; @@ -6540,7 +6540,7 @@ do_setitems(UnpicklerObject *self, Py_ssize_t x) return 0; if ((len - x) % 2 != 0) { PickleState *st = _Pickle_GetGlobalState(); - /* Currupt or hostile pickle -- we never write one like this. */ + /* Corrupt or hostile pickle -- we never write one like this. */ PyErr_SetString(st->UnpicklingError, "odd number of items for SETITEMS"); return -1; diff --git a/Modules/_sre.c b/Modules/_sre.c index 7b21240e9a033..d21b533530187 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -15,7 +15,7 @@ * 2001-05-14 fl fixes for 1.5.2 compatibility * 2001-07-01 fl added BIGCHARSET support (from Martin von Loewis) * 2001-10-18 fl fixed group reset issue (from Matthew Mueller) - * 2001-10-20 fl added split primitive; reenable unicode for 1.6/2.0/2.1 + * 2001-10-20 fl added split primitive; re-enable unicode for 1.6/2.0/2.1 * 2001-10-21 fl added sub/subn primitive * 2001-10-24 fl added finditer primitive (for 2.2 only) * 2001-12-07 fl fixed memory leak in sub/subn (Guido van Rossum) diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 103bd4062292d..5e59549252115 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5269,7 +5269,7 @@ encode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; @@ -5312,7 +5312,7 @@ decode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index 388e26e58d420..813d7ec232c35 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -1215,7 +1215,7 @@ where the corresponding signal handler will be executed.\n\ If *signum* is omitted, SIGINT is assumed.\n\ A subthread can use this function to interrupt the main thread.\n\ \n\ -Note: the default signal hander for SIGINT raises ``KeyboardInterrupt``." +Note: the default signal handler for SIGINT raises ``KeyboardInterrupt``." ); static lockobject *newlockobject(PyObject *module); diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 329b291729d58..2a3e65b6c97df 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -936,7 +936,7 @@ PyTclObject_str(PyTclObject *self) Py_INCREF(self->string); return self->string; } - /* XXX Could chache result if it is non-ASCII. */ + /* XXX Could cache result if it is non-ASCII. */ return unicodeFromTclObj(self->value); } diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 04f6c243b5ca4..90498fb7a7897 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -836,7 +836,7 @@ tracemalloc_clear_filename(void *value) static void tracemalloc_clear_traces(void) { - /* The GIL protects variables againt concurrent access */ + /* The GIL protects variables against concurrent access */ assert(PyGILState_Check()); TABLES_LOCK(); diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index c722330ee491a..04fa09422b213 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -1342,7 +1342,7 @@ tzrule_transitions(_tzrule *rule, int year, int64_t *start, int64_t *end) * could technically be calculated from the timestamp, but given that the * callers of this function already have the year information accessible from * the datetime struct, it is taken as an additional parameter to reduce - * unncessary calculation. + * unnecessary calculation. * */ static _ttinfo * find_tzrule_ttinfo(_tzrule *rule, int64_t ts, unsigned char fold, int year) diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 5ba56eaea6357..034a03c30851a 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -3976,7 +3976,7 @@ initializeEncoding(XML_Parser parser) { const char *s; #ifdef XML_UNICODE char encodingBuf[128]; - /* See comments abount `protoclEncodingName` in parserInit() */ + /* See comments about `protoclEncodingName` in parserInit() */ if (! parser->m_protocolEncodingName) s = NULL; else { diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt index b5b9f7b2e3d4d..6d07cce123643 100644 --- a/Modules/gc_weakref.txt +++ b/Modules/gc_weakref.txt @@ -50,7 +50,7 @@ CT while gc is running. https://www.python.org/sf/1055820 shows how innocent it can be, and also how nasty. Variants of the three -focussed test cases attached to that bug report are now part of Python's +focused test cases attached to that bug report are now part of Python's standard Lib/test/test_gc.py. Jim Fulton gave the best nutshell summary of the new (in 2.4 and 2.3.5) diff --git a/Modules/getpath.c b/Modules/getpath.c index 363d62a0657eb..ef6dd59a084d8 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -1505,7 +1505,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig) } /* If a pyvenv.cfg configure file is found, - argv0_path is overriden with its 'home' variable. */ + argv0_path is overridden with its 'home' variable. */ status = calculate_read_pyenv(calculate); if (_PyStatus_EXCEPTION(status)) { return status; diff --git a/Modules/sha1module.c b/Modules/sha1module.c index 6980051dc08f8..d186aa460a07b 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -269,7 +269,7 @@ sha1_done(struct sha1_state *sha1, unsigned char *out) sha1->curlen = 0; } - /* pad upto 56 bytes of zeroes */ + /* pad up to 56 bytes of zeroes */ while (sha1->curlen < 56) { sha1->buf[sha1->curlen++] = (unsigned char)0; } diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index a4eeec9807c91..5c4ed183facae 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1594,7 +1594,7 @@ signal_get_set_handlers(signal_state_t *state, PyObject *mod_dict) Py_XDECREF(old_func); } - // Instal Python SIGINT handler which raises KeyboardInterrupt + // Install Python SIGINT handler which raises KeyboardInterrupt PyObject* sigint_func = get_handler(SIGINT); if (sigint_func == state->default_handler) { PyObject *int_handler = PyMapping_GetItemString(mod_dict, diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 83f05b72ecbd9..ab8618b341544 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -927,7 +927,7 @@ sock_call_ex(PySocketSockObject *s, reading, but the data then discarded by the OS because of a wrong checksum. - Loop on select() to recheck for socket readyness. */ + Loop on select() to recheck for socket readiness. */ continue; } diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c index cdc94a60a373d..0f2fea15b3a6f 100644 --- a/Modules/syslogmodule.c +++ b/Modules/syslogmodule.c @@ -183,7 +183,7 @@ syslog_syslog(PyObject * self, PyObject * args) PyObject *openargs; /* Continue even if PyTuple_New fails, because openlog(3) is optional. - * So, we can still do loggin in the unlikely event things are so hosed + * So, we can still do logging in the unlikely event things are so hosed * that we can't do this tuple. */ if ((openargs = PyTuple_New(0))) { diff --git a/Modules/zlibmodule.c b/Modules/zlibmodule.c index bc4ff608e81b3..68de701efa45f 100644 --- a/Modules/zlibmodule.c +++ b/Modules/zlibmodule.c @@ -82,7 +82,7 @@ typedef struct { Bytef *next_posi; } _Uint32Window; -/* Initialize the buffer with an inital buffer size. +/* Initialize the buffer with an initial buffer size. On success, return value >= 0 On failure, return value < 0 */ From webhook-mailer at python.org Thu Oct 7 05:17:04 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 07 Oct 2021 09:17:04 -0000 Subject: [Python-checkins] bpo-45041: Restore `sqlite3` executescript behaviour for `SELECT` queries (GH-28509) Message-ID: https://github.com/python/cpython/commit/3f2c433da560d7999a52f9fcba4bbd0898848520 commit: 3f2c433da560d7999a52f9fcba4bbd0898848520 branch: main author: Erlend Egeberg Aasland committer: pablogsal date: 2021-10-07T10:16:45+01:00 summary: bpo-45041: Restore `sqlite3` executescript behaviour for `SELECT` queries (GH-28509) * bpo-45041: Restore sqlite3 executescript behaviour for select queries * Add regression test files: M Lib/sqlite3/test/test_regression.py M Modules/_sqlite/cursor.c diff --git a/Lib/sqlite3/test/test_regression.py b/Lib/sqlite3/test/test_regression.py index ff356734860b6..3d71809d9c11c 100644 --- a/Lib/sqlite3/test/test_regression.py +++ b/Lib/sqlite3/test/test_regression.py @@ -475,6 +475,17 @@ def dup(v): con.execute("drop table t") con.commit() + def test_executescript_step_through_select(self): + with managed_connect(":memory:", in_mem=True) as con: + values = [(v,) for v in range(5)] + with con: + con.execute("create table t(t)") + con.executemany("insert into t values(?)", values) + steps = [] + con.create_function("step", 1, lambda x: steps.append((x,))) + con.executescript("select step(t) from t") + self.assertEqual(steps, values) + if __name__ == "__main__": unittest.main() diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c index 38ccdcf5379d0..ca74a68de4dba 100644 --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -760,7 +760,7 @@ pysqlite_cursor_executescript_impl(pysqlite_Cursor *self, &tail); if (rc == SQLITE_OK) { do { - (void)sqlite3_step(stmt); + rc = sqlite3_step(stmt); } while (rc == SQLITE_ROW); rc = sqlite3_finalize(stmt); } From webhook-mailer at python.org Thu Oct 7 05:26:17 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 09:26:17 -0000 Subject: [Python-checkins] [3.10] Fix typo in whatsnew 3.10.rst (GH-26854) (GH-28782) Message-ID: https://github.com/python/cpython/commit/ee92205bb2b0c3693c5ed32d281d2d3bb40f6e4d commit: ee92205bb2b0c3693c5ed32d281d2d3bb40f6e4d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T02:26:12-07:00 summary: [3.10] Fix typo in whatsnew 3.10.rst (GH-26854) (GH-28782) Thanks for the fix @wiggin15 . (cherry picked from commit 599c07006a636b0a6904008534118a9ba3daf726) Co-authored-by: Arnon Yaari Automerge-Triggered-By: GH:JulienPalard files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 803ba87b047e7..7e8ae42085001 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -1584,7 +1584,7 @@ Deprecated this release, a deprecation warning is raised if the numeric literal is immediately followed by one of keywords :keyword:`and`, :keyword:`else`, :keyword:`for`, :keyword:`if`, :keyword:`in`, :keyword:`is` and :keyword:`or`. - If future releases it will be changed to syntax warning, and finally to + In future releases it will be changed to syntax warning, and finally to syntax error. (Contributed by Serhiy Storchaka in :issue:`43833`). From webhook-mailer at python.org Thu Oct 7 07:47:28 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 07 Oct 2021 11:47:28 -0000 Subject: [Python-checkins] bpo-45400: Fix suggestion test of test_exceptions (GH-28783) Message-ID: https://github.com/python/cpython/commit/4e605666b08b8f863cbbbdaa34bb06988e648d26 commit: 4e605666b08b8f863cbbbdaa34bb06988e648d26 branch: main author: Victor Stinner committer: vstinner date: 2021-10-07T13:47:23+02:00 summary: bpo-45400: Fix suggestion test of test_exceptions (GH-28783) Fix test_name_error_suggestions_do_not_trigger_for_too_many_locals() of test_exceptions if a directory name contains "a1" (like "Python-3.11.0a1"): use a stricter regular expression. files: A Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst M Lib/test/test_exceptions.py diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 289288478c285..85dc3c0f22081 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -1840,7 +1840,7 @@ def f(): with support.captured_stderr() as err: sys.__excepthook__(*sys.exc_info()) - self.assertNotIn("a1", err.getvalue()) + self.assertNotRegex(err.getvalue(), r"NameError.*a1") def test_name_error_with_custom_exceptions(self): def f(): diff --git a/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst b/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst new file mode 100644 index 0000000000000..61b6653320dfc --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst @@ -0,0 +1,3 @@ +Fix test_name_error_suggestions_do_not_trigger_for_too_many_locals() of +test_exceptions if a directory name contains "a1" (like "Python-3.11.0a1"): +use a stricter regular expression. Patch by Victor Stinner. From webhook-mailer at python.org Thu Oct 7 08:11:43 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 12:11:43 -0000 Subject: [Python-checkins] bpo-45400: Fix suggestion test of test_exceptions (GH-28783) Message-ID: https://github.com/python/cpython/commit/d55bf81c4ee2dcdd5dfa3b5a9905a0b32ca8d38b commit: d55bf81c4ee2dcdd5dfa3b5a9905a0b32ca8d38b branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T05:11:38-07:00 summary: bpo-45400: Fix suggestion test of test_exceptions (GH-28783) Fix test_name_error_suggestions_do_not_trigger_for_too_many_locals() of test_exceptions if a directory name contains "a1" (like "Python-3.11.0a1"): use a stricter regular expression. (cherry picked from commit 4e605666b08b8f863cbbbdaa34bb06988e648d26) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst M Lib/test/test_exceptions.py diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 82aa79e996262..143b856280028 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -1840,7 +1840,7 @@ def f(): with support.captured_stderr() as err: sys.__excepthook__(*sys.exc_info()) - self.assertNotIn("a1", err.getvalue()) + self.assertNotRegex(err.getvalue(), r"NameError.*a1") def test_name_error_with_custom_exceptions(self): def f(): diff --git a/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst b/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst new file mode 100644 index 0000000000000..61b6653320dfc --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-07-13-11-45.bpo-45400.h3iT7V.rst @@ -0,0 +1,3 @@ +Fix test_name_error_suggestions_do_not_trigger_for_too_many_locals() of +test_exceptions if a directory name contains "a1" (like "Python-3.11.0a1"): +use a stricter regular expression. Patch by Victor Stinner. From webhook-mailer at python.org Thu Oct 7 09:00:28 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 13:00:28 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Include directory (GH-28745) (GH-28789) Message-ID: https://github.com/python/cpython/commit/f6798391b5d61bb514ec285308a2ccd16b7188dc commit: f6798391b5d61bb514ec285308a2ccd16b7188dc branch: 3.10 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T06:00:23-07:00 summary: [3.10] Fix typos in the Include directory (GH-28745) (GH-28789) (cherry picked from commit 8e8f7522171ef82f2f5049940f815e00e38c6f42) files: M Include/abstract.h M Include/cpython/dictobject.h M Include/cpython/pystate.h M Include/cpython/pytime.h M Include/internal/pycore_traceback.h M Include/object.h diff --git a/Include/abstract.h b/Include/abstract.h index e8d3f9293b98d..9eaab6b2e054c 100644 --- a/Include/abstract.h +++ b/Include/abstract.h @@ -318,7 +318,7 @@ PyAPI_FUNC(int) PyObject_DelItem(PyObject *o, PyObject *key); /* Takes an arbitrary object which must support the (character, single segment) buffer interface and returns a pointer to a read-only memory location - useable as character based input for subsequent processing. + usable as character based input for subsequent processing. Return 0 on success. buffer and buffer_len are only set in case no error occurs. Otherwise, -1 is returned and an exception set. */ diff --git a/Include/cpython/dictobject.h b/Include/cpython/dictobject.h index 6822a65cad95e..641d7bdc48e2e 100644 --- a/Include/cpython/dictobject.h +++ b/Include/cpython/dictobject.h @@ -22,7 +22,7 @@ typedef struct { /* If ma_values is NULL, the table is "combined": keys and values are stored in ma_keys. - If ma_values is not NULL, the table is splitted: + If ma_values is not NULL, the table is split: keys are stored in ma_keys and values are stored in ma_values */ PyObject **ma_values; } PyDictObject; diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index e3ccc54356084..b0d30bd079313 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -242,7 +242,7 @@ PyAPI_FUNC(int) _PyInterpreterState_GetConfigCopy( PyAPI_FUNC(int) _PyInterpreterState_SetConfig( const struct PyConfig *config); -// Get the configuration of the currrent interpreter. +// Get the configuration of the current interpreter. // The caller must hold the GIL. PyAPI_FUNC(const PyConfig*) _Py_GetConfig(void); diff --git a/Include/cpython/pytime.h b/Include/cpython/pytime.h index 56607d199ed54..754c7f4777f9c 100644 --- a/Include/cpython/pytime.h +++ b/Include/cpython/pytime.h @@ -88,13 +88,13 @@ PyAPI_FUNC(_PyTime_t) _PyTime_FromNanoseconds(_PyTime_t ns); PyAPI_FUNC(int) _PyTime_FromNanosecondsObject(_PyTime_t *t, PyObject *obj); -/* Convert a number of seconds (Python float or int) to a timetamp. +/* Convert a number of seconds (Python float or int) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromSecondsObject(_PyTime_t *t, PyObject *obj, _PyTime_round_t round); -/* Convert a number of milliseconds (Python float or int, 10^-3) to a timetamp. +/* Convert a number of milliseconds (Python float or int, 10^-3) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromMillisecondsObject(_PyTime_t *t, PyObject *obj, diff --git a/Include/internal/pycore_traceback.h b/Include/internal/pycore_traceback.h index 4d282308769dc..c01a47639d5e3 100644 --- a/Include/internal/pycore_traceback.h +++ b/Include/internal/pycore_traceback.h @@ -51,7 +51,7 @@ PyAPI_FUNC(void) _Py_DumpTraceback( _PyGILState_GetInterpreterStateUnsafe() in last resort. It is better to pass NULL to interp and current_tstate, the function tries - different options to retrieve these informations. + different options to retrieve this information. This function is signal safe. */ diff --git a/Include/object.h b/Include/object.h index 9e6a8f4656af0..61e638c34973c 100644 --- a/Include/object.h +++ b/Include/object.h @@ -590,7 +590,7 @@ static inline PyObject* _Py_XNewRef(PyObject *obj) } // Py_NewRef() and Py_XNewRef() are exported as functions for the stable ABI. -// Names overriden with macros by static inline functions for best +// Names overridden with macros by static inline functions for best // performances. #define Py_NewRef(obj) _Py_NewRef(_PyObject_CAST(obj)) #define Py_XNewRef(obj) _Py_XNewRef(_PyObject_CAST(obj)) From webhook-mailer at python.org Thu Oct 7 09:01:10 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 13:01:10 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Include directory (GH-28745) (GH-28788) Message-ID: https://github.com/python/cpython/commit/6c97b03cf64176da96aa68b2281f1cc702fb96a1 commit: 6c97b03cf64176da96aa68b2281f1cc702fb96a1 branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T06:01:05-07:00 summary: [3.9] Fix typos in the Include directory (GH-28745) (GH-28788) (cherry picked from commit 8e8f7522171ef82f2f5049940f815e00e38c6f42) files: M Include/abstract.h M Include/cpython/dictobject.h M Include/internal/pycore_traceback.h M Include/pytime.h diff --git a/Include/abstract.h b/Include/abstract.h index bb51c668ac698..d31d66e6b4aea 100644 --- a/Include/abstract.h +++ b/Include/abstract.h @@ -318,7 +318,7 @@ PyAPI_FUNC(int) PyObject_DelItem(PyObject *o, PyObject *key); /* Takes an arbitrary object which must support the (character, single segment) buffer interface and returns a pointer to a read-only memory location - useable as character based input for subsequent processing. + usable as character based input for subsequent processing. Return 0 on success. buffer and buffer_len are only set in case no error occurs. Otherwise, -1 is returned and an exception set. */ diff --git a/Include/cpython/dictobject.h b/Include/cpython/dictobject.h index e33a0d156fead..50f4c4a9e5502 100644 --- a/Include/cpython/dictobject.h +++ b/Include/cpython/dictobject.h @@ -26,7 +26,7 @@ typedef struct { /* If ma_values is NULL, the table is "combined": keys and values are stored in ma_keys. - If ma_values is not NULL, the table is splitted: + If ma_values is not NULL, the table is split: keys are stored in ma_keys and values are stored in ma_values */ PyObject **ma_values; } PyDictObject; diff --git a/Include/internal/pycore_traceback.h b/Include/internal/pycore_traceback.h index 1f092411a72ba..c23290ebec599 100644 --- a/Include/internal/pycore_traceback.h +++ b/Include/internal/pycore_traceback.h @@ -51,7 +51,7 @@ PyAPI_FUNC(void) _Py_DumpTraceback( _PyGILState_GetInterpreterStateUnsafe() in last resort. It is better to pass NULL to interp and current_tstate, the function tries - different options to retrieve these informations. + different options to retrieve this information. This function is signal safe. */ diff --git a/Include/pytime.h b/Include/pytime.h index bdda1da2e6b8f..c79eba2f5bc6f 100644 --- a/Include/pytime.h +++ b/Include/pytime.h @@ -91,13 +91,13 @@ PyAPI_FUNC(_PyTime_t) _PyTime_FromNanoseconds(_PyTime_t ns); PyAPI_FUNC(int) _PyTime_FromNanosecondsObject(_PyTime_t *t, PyObject *obj); -/* Convert a number of seconds (Python float or int) to a timetamp. +/* Convert a number of seconds (Python float or int) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromSecondsObject(_PyTime_t *t, PyObject *obj, _PyTime_round_t round); -/* Convert a number of milliseconds (Python float or int, 10^-3) to a timetamp. +/* Convert a number of milliseconds (Python float or int, 10^-3) to a timestamp. Raise an exception and return -1 on error, return 0 on success. */ PyAPI_FUNC(int) _PyTime_FromMillisecondsObject(_PyTime_t *t, PyObject *obj, From webhook-mailer at python.org Thu Oct 7 09:17:12 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 13:17:12 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Modules directory (GH-28761). (GH-28791) Message-ID: https://github.com/python/cpython/commit/8f762349a20672f298584e5134f3f8b1b307745a commit: 8f762349a20672f298584e5134f3f8b1b307745a branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T06:17:05-07:00 summary: [3.9] Fix typos in the Modules directory (GH-28761). (GH-28791) (cherry picked from commit dd02a696e55b450413e765e698e653d781ca4205) Co-authored-by: Christian Clauss Automerge-Triggered-By: GH:JulienPalard files: M Modules/_ctypes/callproc.c M Modules/_ctypes/cfield.c M Modules/_ctypes/ctypes.h M Modules/_pickle.c M Modules/_sre.c M Modules/_testcapimodule.c M Modules/_tkinter.c M Modules/_tracemalloc.c M Modules/_zoneinfo.c M Modules/expat/xmlparse.c M Modules/gc_weakref.txt M Modules/getpath.c M Modules/sha1module.c M Modules/socketmodule.c M Modules/syslogmodule.c diff --git a/Modules/_ctypes/callproc.c b/Modules/_ctypes/callproc.c index dafc51e5d3db7..9aff890afe9de 100644 --- a/Modules/_ctypes/callproc.c +++ b/Modules/_ctypes/callproc.c @@ -838,7 +838,7 @@ static int _call_function_pointer(int flags, # define HAVE_FFI_PREP_CIF_VAR_RUNTIME false # endif - /* Even on Apple-arm64 the calling convention for variadic functions conincides + /* Even on Apple-arm64 the calling convention for variadic functions coincides * with the standard calling convention in the case that the function called * only with its fixed arguments. Thus, we do not need a special flag to be * set on variadic functions. We treat a function as variadic if it is called diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 7ebd4ba76afb7..06b98a6d66fc3 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -35,7 +35,7 @@ PyCField_new(PyTypeObject *type, PyObject *args, PyObject *kwds) * Expects the size, index and offset for the current field in *psize and * *poffset, stores the total size so far in *psize, the offset for the next * field in *poffset, the alignment requirements for the current field in - * *palign, and returns a field desriptor for this field. + * *palign, and returns a field descriptor for this field. */ /* * bitfields extension: diff --git a/Modules/_ctypes/ctypes.h b/Modules/_ctypes/ctypes.h index 3f20031d671a8..0f422113a89cb 100644 --- a/Modules/_ctypes/ctypes.h +++ b/Modules/_ctypes/ctypes.h @@ -208,7 +208,7 @@ typedef struct { PyObject *checker; int flags; /* calling convention and such */ - /* pep3118 fields, pointers neeed PyMem_Free */ + /* pep3118 fields, pointers need PyMem_Free */ char *format; int ndim; Py_ssize_t *shape; diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 8dea2c6ea0c20..ff9402bd1606e 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4531,7 +4531,7 @@ dump(PicklerObject *self, PyObject *obj) * call when setting the reducer_override attribute of the Pickler instance * to a bound method of the same instance. This is important as the Pickler * instance holds a reference to each object it has pickled (through its - * memo): thus, these objects wont be garbage-collected as long as the + * memo): thus, these objects won't be garbage-collected as long as the * Pickler itself is not collected. */ Py_CLEAR(self->reducer_override); return status; @@ -6587,7 +6587,7 @@ do_setitems(UnpicklerObject *self, Py_ssize_t x) return 0; if ((len - x) % 2 != 0) { PickleState *st = _Pickle_GetGlobalState(); - /* Currupt or hostile pickle -- we never write one like this. */ + /* Corrupt or hostile pickle -- we never write one like this. */ PyErr_SetString(st->UnpicklingError, "odd number of items for SETITEMS"); return -1; diff --git a/Modules/_sre.c b/Modules/_sre.c index bdc427822d7e1..8225c36da1a38 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -15,7 +15,7 @@ * 2001-05-14 fl fixes for 1.5.2 compatibility * 2001-07-01 fl added BIGCHARSET support (from Martin von Loewis) * 2001-10-18 fl fixed group reset issue (from Matthew Mueller) - * 2001-10-20 fl added split primitive; reenable unicode for 1.6/2.0/2.1 + * 2001-10-20 fl added split primitive; re-enable unicode for 1.6/2.0/2.1 * 2001-10-21 fl added sub/subn primitive * 2001-10-24 fl added finditer primitive (for 2.2 only) * 2001-12-07 fl fixed memory leak in sub/subn (Guido van Rossum) diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index f6558caee9e34..ae1284178dc60 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -5127,7 +5127,7 @@ encode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; @@ -5170,7 +5170,7 @@ decode_locale_ex(PyObject *self, PyObject *args) PyErr_SetString(PyExc_ValueError, "unsupported error handler"); break; default: - PyErr_SetString(PyExc_ValueError, "unknow error code"); + PyErr_SetString(PyExc_ValueError, "unknown error code"); break; } return res; diff --git a/Modules/_tkinter.c b/Modules/_tkinter.c index 19853ce491182..e153047b778b7 100644 --- a/Modules/_tkinter.c +++ b/Modules/_tkinter.c @@ -936,7 +936,7 @@ PyTclObject_str(PyTclObject *self) Py_INCREF(self->string); return self->string; } - /* XXX Could chache result if it is non-ASCII. */ + /* XXX Could cache result if it is non-ASCII. */ return unicodeFromTclObj(self->value); } diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 04f6c243b5ca4..90498fb7a7897 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -836,7 +836,7 @@ tracemalloc_clear_filename(void *value) static void tracemalloc_clear_traces(void) { - /* The GIL protects variables againt concurrent access */ + /* The GIL protects variables against concurrent access */ assert(PyGILState_Check()); TABLES_LOCK(); diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index f655768496e13..d7945d31affea 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -1347,7 +1347,7 @@ tzrule_transitions(_tzrule *rule, int year, int64_t *start, int64_t *end) * could technically be calculated from the timestamp, but given that the * callers of this function already have the year information accessible from * the datetime struct, it is taken as an additional parameter to reduce - * unncessary calculation. + * unnecessary calculation. * */ static _ttinfo * find_tzrule_ttinfo(_tzrule *rule, int64_t ts, unsigned char fold, int year) diff --git a/Modules/expat/xmlparse.c b/Modules/expat/xmlparse.c index 5ba56eaea6357..034a03c30851a 100644 --- a/Modules/expat/xmlparse.c +++ b/Modules/expat/xmlparse.c @@ -3976,7 +3976,7 @@ initializeEncoding(XML_Parser parser) { const char *s; #ifdef XML_UNICODE char encodingBuf[128]; - /* See comments abount `protoclEncodingName` in parserInit() */ + /* See comments about `protoclEncodingName` in parserInit() */ if (! parser->m_protocolEncodingName) s = NULL; else { diff --git a/Modules/gc_weakref.txt b/Modules/gc_weakref.txt index b5b9f7b2e3d4d..6d07cce123643 100644 --- a/Modules/gc_weakref.txt +++ b/Modules/gc_weakref.txt @@ -50,7 +50,7 @@ CT while gc is running. https://www.python.org/sf/1055820 shows how innocent it can be, and also how nasty. Variants of the three -focussed test cases attached to that bug report are now part of Python's +focused test cases attached to that bug report are now part of Python's standard Lib/test/test_gc.py. Jim Fulton gave the best nutshell summary of the new (in 2.4 and 2.3.5) diff --git a/Modules/getpath.c b/Modules/getpath.c index 728ecad052e14..5dbe57c95093a 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -1510,7 +1510,7 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig) } /* If a pyvenv.cfg configure file is found, - argv0_path is overriden with its 'home' variable. */ + argv0_path is overridden with its 'home' variable. */ status = calculate_read_pyenv(calculate); if (_PyStatus_EXCEPTION(status)) { return status; diff --git a/Modules/sha1module.c b/Modules/sha1module.c index 9c75cc99dba77..44fe8ef7b356f 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -269,7 +269,7 @@ sha1_done(struct sha1_state *sha1, unsigned char *out) sha1->curlen = 0; } - /* pad upto 56 bytes of zeroes */ + /* pad up to 56 bytes of zeroes */ while (sha1->curlen < 56) { sha1->buf[sha1->curlen++] = (unsigned char)0; } diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 0ac7fd0a607a0..a22060d399082 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -965,7 +965,7 @@ sock_call_ex(PySocketSockObject *s, reading, but the data then discarded by the OS because of a wrong checksum. - Loop on select() to recheck for socket readyness. */ + Loop on select() to recheck for socket readiness. */ continue; } diff --git a/Modules/syslogmodule.c b/Modules/syslogmodule.c index cdc94a60a373d..0f2fea15b3a6f 100644 --- a/Modules/syslogmodule.c +++ b/Modules/syslogmodule.c @@ -183,7 +183,7 @@ syslog_syslog(PyObject * self, PyObject * args) PyObject *openargs; /* Continue even if PyTuple_New fails, because openlog(3) is optional. - * So, we can still do loggin in the unlikely event things are so hosed + * So, we can still do logging in the unlikely event things are so hosed * that we can't do this tuple. */ if ((openargs = PyTuple_New(0))) { From webhook-mailer at python.org Thu Oct 7 09:42:16 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 13:42:16 -0000 Subject: [Python-checkins] [3.9] [Misc] [Mac] Fix typos found using codespell (GH-28756) (GH-28793) Message-ID: https://github.com/python/cpython/commit/71d56a72313d66e73ca1b61130d2e617235c8a93 commit: 71d56a72313d66e73ca1b61130d2e617235c8a93 branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T06:42:07-07:00 summary: [3.9] [Misc] [Mac] Fix typos found using codespell (GH-28756) (GH-28793) (cherry picked from commit 470145f572b53fe73518cda1eeacc56fec78c1b2) Automerge-Triggered-By: GH:JulienPalard files: M Mac/BuildScript/build-installer.py M Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py M Mac/PythonLauncher/MyAppDelegate.m M Mac/README.rst M Misc/NEWS.d/3.9.0a1.rst M Misc/NEWS.d/3.9.0a3.rst M Misc/NEWS.d/3.9.0a5.rst M Misc/NEWS.d/3.9.1rc1.rst diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index cbb6cab9e3563..bd1c4d4fe6d8d 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -443,7 +443,7 @@ def pkg_recipes(): source="/pydocs", readme="""\ This package installs the python documentation at a location - that is useable for pydoc and IDLE. + that is usable for pydoc and IDLE. """, postflight="scripts/postflight.documentation", required=False, @@ -1589,7 +1589,7 @@ def buildDMG(): # instead of 11. We should not run into that situation here.) # Also we should use "macos" instead of "macosx" going forward. # - # To maintain compability for legacy variants, the file name for + # To maintain compatibility for legacy variants, the file name for # builds on macOS 10.15 and earlier remains: # python-3.x.y-macosx10.z.{dmg->pkg} # e.g. python-3.9.4-macosx10.9.{dmg->pkg} diff --git a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py index 5994c18ff876b..ccc18d4aee438 100644 --- a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py +++ b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py @@ -35,7 +35,7 @@ # idlemain.py running under the symlinked python. # This is the magic step. # 4. During interpreter initialization, because PYTHONEXECUTABLE is defined, -# sys.executable may get set to an unuseful value. +# sys.executable may get set to an useless value. # # (Note that the IDLE script and the setting of PYTHONEXECUTABLE is # generated automatically by bundlebuilder in the Python 2.x build. diff --git a/Mac/PythonLauncher/MyAppDelegate.m b/Mac/PythonLauncher/MyAppDelegate.m index 25779a2540a37..9cc2aa0ad9098 100644 --- a/Mac/PythonLauncher/MyAppDelegate.m +++ b/Mac/PythonLauncher/MyAppDelegate.m @@ -22,7 +22,7 @@ - (void)applicationDidFinishLaunching:(NSNotification *)notification { // Test that the file mappings are correct [self testFileTypeBinding]; - // If we were opened because of a file drag or doubleclick + // If we were opened because of a file drag or double-click // we've set initial_action_done in shouldShowUI // Otherwise we open a preferences dialog. if (!initial_action_done) { diff --git a/Mac/README.rst b/Mac/README.rst index 1670fe5747aad..f7c4df085de7a 100644 --- a/Mac/README.rst +++ b/Mac/README.rst @@ -296,7 +296,7 @@ How do I create a binary distribution? Download and unpack the source release from https://www.python.org/download/. Go to the directory ``Mac/BuildScript``. There you will find a script ``build-installer.py`` that does all the work. This will download and build -a number of 3rd-party libaries, configures and builds a framework Python, +a number of 3rd-party libraries, configures and builds a framework Python, installs it, creates the installer package files and then packs this in a DMG image. The script also builds an HTML copy of the current Python documentation set for this release for inclusion in the framework. The diff --git a/Misc/NEWS.d/3.9.0a1.rst b/Misc/NEWS.d/3.9.0a1.rst index 0a6a6eb287145..a9b6694c133f1 100644 --- a/Misc/NEWS.d/3.9.0a1.rst +++ b/Misc/NEWS.d/3.9.0a1.rst @@ -279,7 +279,7 @@ visited by ``tp_traverse()`` are valid. .. nonce: Xgc6F_ .. section: Core and Builtins -Remove unecessary intersection and update set operation in dictview with +Remove unnecessary intersection and update set operation in dictview with empty set. (Contributed by Dong-hee Na in :issue:`38210`.) .. @@ -868,7 +868,7 @@ Fix the :c:func:`PySys_Audit` call in :class:`mmap.mmap`. .. nonce: WJkgKV .. section: Core and Builtins -Remove an unnecssary Py_XINCREF in classobject.c. +Remove an unnecessary Py_XINCREF in classobject.c. .. @@ -1224,7 +1224,7 @@ Anthony Sottile. .. nonce: cyq5nr .. section: Library -Now :func:`~logging.config.fileConfig` correcty sets the .name of handlers +Now :func:`~logging.config.fileConfig` correctly sets the .name of handlers loaded. .. @@ -1637,7 +1637,7 @@ Preserve subclassing in inspect.Signature.from_callable. .. nonce: nHAbuJ .. section: Library -Names of hashing algorithms frome OpenSSL are now normalized to follow +Names of hashing algorithms from OpenSSL are now normalized to follow Python's naming conventions. For example OpenSSL uses sha3-512 instead of sha3_512 or blake2b512 instead of blake2b. @@ -2403,7 +2403,7 @@ Fixed comparisons of :class:`datetime.timedelta` and .. nonce: 7UV5d0 .. section: Library -Syncronize ``importlib.metadata`` with `importlib_metadata 0.19 +Synchronize ``importlib.metadata`` with `importlib_metadata 0.19 `_, improving handling of EGG-INFO files and fixing a crash when entry point names contained colons. @@ -3002,7 +3002,7 @@ on platforms with 16-bit :c:type:`wchar_t` (for example, Windows and AIX). In a subinterpreter, spawning a daemon thread now raises an exception. Daemon threads were never supported in subinterpreters. Previously, the -subinterpreter finalization crashed with a Pyton fatal error if a daemon +subinterpreter finalization crashed with a Python fatal error if a daemon thread was still running. .. @@ -3065,7 +3065,7 @@ internal tasks weak set is changed by another thread during iteration. .. section: Library :class:`_pyio.IOBase` destructor now does nothing if getting the ``closed`` -attribute fails to better mimick :class:`_io.IOBase` finalizer. +attribute fails to better mimic :class:`_io.IOBase` finalizer. .. @@ -4038,7 +4038,7 @@ crypto policies. Use PKCS#8 format with AES256 encryption instead. .. nonce: _x-9uH .. section: Tests -test.support now has a helper function to check for availibility of a hash +test.support now has a helper function to check for availability of a hash digest function. Several tests are refactored avoid MD5 and use SHA256 instead. Other tests are marked to use MD5 and skipped when MD5 is disabled. diff --git a/Misc/NEWS.d/3.9.0a3.rst b/Misc/NEWS.d/3.9.0a3.rst index 6c71d7e839d05..77ccc7453c215 100644 --- a/Misc/NEWS.d/3.9.0a3.rst +++ b/Misc/NEWS.d/3.9.0a3.rst @@ -178,7 +178,7 @@ last iteration of asynchronous for loops. Patch by Pablo Galindo. .. nonce: WG9alt .. section: Core and Builtins -Fix incorrent line execution reporting in trace functions when tracing +Fix incorrect line execution reporting in trace functions when tracing exception handlers with name binding. Patch by Pablo Galindo. .. @@ -685,7 +685,7 @@ but not required. Patch by Juergen Gmach. .. section: Library Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result -upon inheritence. Patch by Bar Harel. +upon inheritance. Patch by Bar Harel. .. diff --git a/Misc/NEWS.d/3.9.0a5.rst b/Misc/NEWS.d/3.9.0a5.rst index ede281b7a9b1b..5e0a50ce719cf 100644 --- a/Misc/NEWS.d/3.9.0a5.rst +++ b/Misc/NEWS.d/3.9.0a5.rst @@ -175,7 +175,7 @@ convention. Patch by Dong-hee Na. .. nonce: FE9S21 .. section: Core and Builtins -Chaged list overallocation strategy. It no longer overallocates if the new +Changed list overallocation strategy. It no longer overallocates if the new size is closer to overalocated size than to the old size and adds padding. .. diff --git a/Misc/NEWS.d/3.9.1rc1.rst b/Misc/NEWS.d/3.9.1rc1.rst index 51edf2fed7e83..279ad12962485 100644 --- a/Misc/NEWS.d/3.9.1rc1.rst +++ b/Misc/NEWS.d/3.9.1rc1.rst @@ -317,7 +317,7 @@ webbrowser: Ignore *NotADirectoryError* when calling ``xdg-settings``. .. nonce: 6aDbty .. section: Library -``binhex.binhex()`` consisently writes macOS 9 line endings. +``binhex.binhex()`` consistently writes macOS 9 line endings. .. From webhook-mailer at python.org Thu Oct 7 10:09:49 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 14:09:49 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Objects directory (GH-28766) (GH-28795) Message-ID: https://github.com/python/cpython/commit/960e7b3ba13b26f45775551ab6f3c062b8666b74 commit: 960e7b3ba13b26f45775551ab6f3c062b8666b74 branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T07:09:41-07:00 summary: [3.9] Fix typos in the Objects directory (GH-28766) (GH-28795) (cherry picked from commit 5f401f10400123afa9171548c432ea3fc37c0736) Automerge-Triggered-By: GH:JulienPalard files: M Objects/floatobject.c M Objects/listobject.c M Objects/listsort.txt M Objects/obmalloc.c M Objects/setobject.c M Objects/unicodeobject.c M PC/getpathp.c diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 6adc90ab9a2c3..8538a051b1982 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -2291,7 +2291,7 @@ _PyFloat_Pack8(double x, unsigned char *p, int le) flo = 0; ++fhi; if (fhi >> 28) { - /* And it also progagated out of the next 28 bits. */ + /* And it also propagated out of the next 28 bits. */ fhi = 0; ++e; if (e >= 2047) diff --git a/Objects/listobject.c b/Objects/listobject.c index 6c94ba57cfe8e..1e868b43c0980 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2012,7 +2012,7 @@ safe_object_compare(PyObject *v, PyObject *w, MergeState *ms) return PyObject_RichCompareBool(v, w, Py_LT); } -/* Homogeneous compare: safe for any two compareable objects of the same type. +/* Homogeneous compare: safe for any two comparable objects of the same type. * (ms->key_richcompare is set to ob_type->tp_richcompare in the * pre-sort check.) */ diff --git a/Objects/listsort.txt b/Objects/listsort.txt index 174777a2658dc..42809d71bc44a 100644 --- a/Objects/listsort.txt +++ b/Objects/listsort.txt @@ -758,6 +758,6 @@ OPTIMIZATION OF INDIVIDUAL COMPARISONS As noted above, even the simplest Python comparison triggers a large pile of C-level pointer dereferences, conditionals, and function calls. This can be partially mitigated by pre-scanning the data to determine whether the data is -homogenous with respect to type. If so, it is sometimes possible to +homogeneous with respect to type. If so, it is sometimes possible to substitute faster type-specific comparisons for the slower, generic PyObject_RichCompareBool. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index eb34f10bddf99..9f8e0d114fff8 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -836,7 +836,7 @@ static int running_on_valgrind = -1; /* * Alignment of addresses returned to the user. 8-bytes alignment works - * on most current architectures (with 32-bit or 64-bit address busses). + * on most current architectures (with 32-bit or 64-bit address buses). * The alignment value is also used for grouping small requests in size * classes spaced ALIGNMENT bytes apart. * diff --git a/Objects/setobject.c b/Objects/setobject.c index 76b1944db4558..4bd5777f967da 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1410,7 +1410,7 @@ set_difference_update_internal(PySetObject *so, PyObject *other) /* Optimization: When the other set is more than 8 times larger than the base set, replace the other set with - interesection of the two sets. + intersection of the two sets. */ if ((PySet_GET_SIZE(other) >> 3) > PySet_GET_SIZE(so)) { other = set_intersection(so, other); diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 4fb06f9fd26e0..38fb3ffc5eb37 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7335,7 +7335,7 @@ PyUnicode_AsASCIIString(PyObject *unicode) #endif /* INT_MAX is the theoretical largest chunk (or INT_MAX / 2 when - transcoding from UTF-16), but INT_MAX / 4 perfoms better in + transcoding from UTF-16), but INT_MAX / 4 performs better in both cases also and avoids partial characters overrunning the length limit in MultiByteToWideChar on Windows */ #define DECODING_CHUNK_SIZE (INT_MAX/4) @@ -16121,7 +16121,7 @@ init_fs_codec(PyInterpreterState *interp) _Py_error_handler error_handler; error_handler = get_error_handler_wide(config->filesystem_errors); if (error_handler == _Py_ERROR_UNKNOWN) { - PyErr_SetString(PyExc_RuntimeError, "unknow filesystem error handler"); + PyErr_SetString(PyExc_RuntimeError, "unknown filesystem error handler"); return -1; } diff --git a/PC/getpathp.c b/PC/getpathp.c index 603a1eb13c4ff..dc5b201d145f3 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -332,7 +332,7 @@ extern const char *PyWin_DLLVersionString; Returns NULL, or a pointer that should be freed. XXX - this code is pretty strange, as it used to also - work on Win16, where the buffer sizes werent available + work on Win16, where the buffer sizes were not available in advance. It could be simplied now Win16/Win32s is dead! */ static wchar_t * From webhook-mailer at python.org Thu Oct 7 10:31:49 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 14:31:49 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Objects directory (GH-28766) (GH-28797) Message-ID: https://github.com/python/cpython/commit/dcfbe4f72d927a3f63a9df9e7add061c71fce547 commit: dcfbe4f72d927a3f63a9df9e7add061c71fce547 branch: 3.10 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T07:31:33-07:00 summary: [3.10] Fix typos in the Objects directory (GH-28766) (GH-28797) (cherry picked from commit 5f401f10400123afa9171548c432ea3fc37c0736) Automerge-Triggered-By: GH:JulienPalard files: M Objects/floatobject.c M Objects/listobject.c M Objects/listsort.txt M Objects/obmalloc.c M Objects/setobject.c M Objects/unicodeobject.c M PC/getpathp.c M Parser/tokenizer.h diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 92faa7c132023..e4ce7e74d2c57 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -2350,7 +2350,7 @@ _PyFloat_Pack8(double x, unsigned char *p, int le) flo = 0; ++fhi; if (fhi >> 28) { - /* And it also progagated out of the next 28 bits. */ + /* And it also propagated out of the next 28 bits. */ fhi = 0; ++e; if (e >= 2047) diff --git a/Objects/listobject.c b/Objects/listobject.c index 898cbc20c5f81..533ee7436d311 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -2024,7 +2024,7 @@ safe_object_compare(PyObject *v, PyObject *w, MergeState *ms) return PyObject_RichCompareBool(v, w, Py_LT); } -/* Homogeneous compare: safe for any two compareable objects of the same type. +/* Homogeneous compare: safe for any two comparable objects of the same type. * (ms->key_richcompare is set to ob_type->tp_richcompare in the * pre-sort check.) */ diff --git a/Objects/listsort.txt b/Objects/listsort.txt index 174777a2658dc..42809d71bc44a 100644 --- a/Objects/listsort.txt +++ b/Objects/listsort.txt @@ -758,6 +758,6 @@ OPTIMIZATION OF INDIVIDUAL COMPARISONS As noted above, even the simplest Python comparison triggers a large pile of C-level pointer dereferences, conditionals, and function calls. This can be partially mitigated by pre-scanning the data to determine whether the data is -homogenous with respect to type. If so, it is sometimes possible to +homogeneous with respect to type. If so, it is sometimes possible to substitute faster type-specific comparisons for the slower, generic PyObject_RichCompareBool. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 1e06bee5c50ff..15c442b858d5f 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -836,7 +836,7 @@ static int running_on_valgrind = -1; /* * Alignment of addresses returned to the user. 8-bytes alignment works - * on most current architectures (with 32-bit or 64-bit address busses). + * on most current architectures (with 32-bit or 64-bit address buses). * The alignment value is also used for grouping small requests in size * classes spaced ALIGNMENT bytes apart. * diff --git a/Objects/setobject.c b/Objects/setobject.c index 9d4cfd35e328f..6524963b8abc3 100644 --- a/Objects/setobject.c +++ b/Objects/setobject.c @@ -1407,7 +1407,7 @@ set_difference_update_internal(PySetObject *so, PyObject *other) /* Optimization: When the other set is more than 8 times larger than the base set, replace the other set with - interesection of the two sets. + intersection of the two sets. */ if ((PySet_GET_SIZE(other) >> 3) > PySet_GET_SIZE(so)) { other = set_intersection(so, other); diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 849dd5cfe2f10..11a3a6d0a4dce 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -7473,7 +7473,7 @@ PyUnicode_AsASCIIString(PyObject *unicode) #endif /* INT_MAX is the theoretical largest chunk (or INT_MAX / 2 when - transcoding from UTF-16), but INT_MAX / 4 perfoms better in + transcoding from UTF-16), but INT_MAX / 4 performs better in both cases also and avoids partial characters overrunning the length limit in MultiByteToWideChar on Windows */ #define DECODING_CHUNK_SIZE (INT_MAX/4) @@ -16149,7 +16149,7 @@ init_fs_codec(PyInterpreterState *interp) _Py_error_handler error_handler; error_handler = get_error_handler_wide(config->filesystem_errors); if (error_handler == _Py_ERROR_UNKNOWN) { - PyErr_SetString(PyExc_RuntimeError, "unknow filesystem error handler"); + PyErr_SetString(PyExc_RuntimeError, "unknown filesystem error handler"); return -1; } diff --git a/PC/getpathp.c b/PC/getpathp.c index 603a1eb13c4ff..dc5b201d145f3 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -332,7 +332,7 @@ extern const char *PyWin_DLLVersionString; Returns NULL, or a pointer that should be freed. XXX - this code is pretty strange, as it used to also - work on Win16, where the buffer sizes werent available + work on Win16, where the buffer sizes were not available in advance. It could be simplied now Win16/Win32s is dead! */ static wchar_t * diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h index a40f7d9687b44..677f9dba490be 100644 --- a/Parser/tokenizer.h +++ b/Parser/tokenizer.h @@ -23,7 +23,7 @@ enum interactive_underflow_t { /* Normal mode of operation: return a new token when asked in interactie mode */ IUNDERFLOW_NORMAL, /* Forcefully return ENDMARKER when asked for a new token in interactive mode. This - * can be used to prevent the tokenizer to promt the user for new tokens */ + * can be used to prevent the tokenizer to prompt the user for new tokens */ IUNDERFLOW_STOP, }; From webhook-mailer at python.org Thu Oct 7 10:36:48 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 14:36:48 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Python directory (GH-28767) (GH-28798) Message-ID: https://github.com/python/cpython/commit/78efc9aff254ec7e7a7b6512293674899e5d3615 commit: 78efc9aff254ec7e7a7b6512293674899e5d3615 branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T07:36:39-07:00 summary: [3.9] Fix typos in the Python directory (GH-28767) (GH-28798) (cherry picked from commit db693df3e112c5a61f2cbef63eedce3a36520ded) Automerge-Triggered-By: GH:JulienPalard files: M Python/fileutils.c M Python/initconfig.c M Python/pathconfig.c M Python/pythonrun.c diff --git a/Python/fileutils.c b/Python/fileutils.c index d072363535c80..26e97df4ef346 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -223,7 +223,7 @@ check_force_ascii(void) ch = (unsigned char)0xA7; res = _Py_mbstowcs(&wch, (char*)&ch, 1); if (res != DECODE_ERROR && wch == L'\xA7') { - /* On HP-UX withe C locale or the POSIX locale, + /* On HP-UX with C locale or the POSIX locale, nl_langinfo(CODESET) announces "roman8", whereas mbstowcs() uses Latin1 encoding in practice. Force ASCII in this case. diff --git a/Python/initconfig.c b/Python/initconfig.c index 3caed385ef6af..116ee33fee113 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -2135,7 +2135,7 @@ warnoptions_append(PyConfig *config, PyWideStringList *options, { /* config_init_warnoptions() add existing config warnoptions at the end: ensure that the new option is not already present in this list to - prevent change the options order whne config_init_warnoptions() is + prevent change the options order when config_init_warnoptions() is called twice. */ if (_PyWideStringList_Find(&config->warnoptions, option)) { /* Already present: do nothing */ diff --git a/Python/pathconfig.c b/Python/pathconfig.c index 9a302213e77b6..b2ce86e64ce8c 100644 --- a/Python/pathconfig.c +++ b/Python/pathconfig.c @@ -331,7 +331,7 @@ config_calculate_pathconfig(PyConfig *config) #undef COPY_ATTR #ifdef MS_WINDOWS - /* If a ._pth file is found: isolated and site_import are overriden */ + /* If a ._pth file is found: isolated and site_import are overridden */ if (pathconfig.isolated != -1) { config->isolated = pathconfig.isolated; } diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 8cdbb3790fbcf..50c59b69f31fc 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -2,7 +2,7 @@ /* Top level execution of Python code (including in __main__) */ /* To help control the interfaces between the startup, execution and - * shutdown code, the phases are split across separate modules (boostrap, + * shutdown code, the phases are split across separate modules (bootstrap, * pythonrun, shutdown) */ From webhook-mailer at python.org Thu Oct 7 11:09:43 2021 From: webhook-mailer at python.org (JulienPalard) Date: Thu, 07 Oct 2021 15:09:43 -0000 Subject: [Python-checkins] [3.10] [Misc] [Mac] Fix typos found using codespell (GH-28756) (GH-28794) Message-ID: https://github.com/python/cpython/commit/212140dceb72d40ba19e4aef089c707dbbcfd5cb commit: 212140dceb72d40ba19e4aef089c707dbbcfd5cb branch: 3.10 author: Christian Clauss committer: JulienPalard date: 2021-10-07T17:09:30+02:00 summary: [3.10] [Misc] [Mac] Fix typos found using codespell (GH-28756) (GH-28794) (cherry picked from commit 470145f572b53fe73518cda1eeacc56fec78c1b2) files: M Mac/BuildScript/build-installer.py M Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py M Mac/PythonLauncher/MyAppDelegate.m M Mac/README.rst M Misc/NEWS.d/3.10.0a1.rst M Misc/NEWS.d/3.10.0a2.rst M Misc/NEWS.d/3.10.0a3.rst M Misc/NEWS.d/3.10.0a4.rst M Misc/NEWS.d/3.10.0a5.rst M Misc/NEWS.d/3.10.0a6.rst M Misc/NEWS.d/3.10.0a7.rst M Misc/NEWS.d/3.10.0b1.rst M Misc/NEWS.d/3.9.0a1.rst M Misc/NEWS.d/3.9.0a3.rst M Misc/NEWS.d/3.9.0a5.rst M Misc/stable_abi.txt diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index e89b1c1a42100..51c8523a8880b 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -454,7 +454,7 @@ def pkg_recipes(): source="/pydocs", readme="""\ This package installs the python documentation at a location - that is useable for pydoc and IDLE. + that is usable for pydoc and IDLE. """, postflight="scripts/postflight.documentation", required=False, @@ -1602,7 +1602,7 @@ def buildDMG(): # instead of 11. We should not run into that situation here.) # Also we should use "macos" instead of "macosx" going forward. # - # To maintain compability for legacy variants, the file name for + # To maintain compatibility for legacy variants, the file name for # builds on macOS 10.15 and earlier remains: # python-3.x.y-macosx10.z.{dmg->pkg} # e.g. python-3.9.4-macosx10.9.{dmg->pkg} diff --git a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py index 5994c18ff876b..ccc18d4aee438 100644 --- a/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py +++ b/Mac/IDLE/IDLE.app/Contents/Resources/idlemain.py @@ -35,7 +35,7 @@ # idlemain.py running under the symlinked python. # This is the magic step. # 4. During interpreter initialization, because PYTHONEXECUTABLE is defined, -# sys.executable may get set to an unuseful value. +# sys.executable may get set to an useless value. # # (Note that the IDLE script and the setting of PYTHONEXECUTABLE is # generated automatically by bundlebuilder in the Python 2.x build. diff --git a/Mac/PythonLauncher/MyAppDelegate.m b/Mac/PythonLauncher/MyAppDelegate.m index 25779a2540a37..9cc2aa0ad9098 100644 --- a/Mac/PythonLauncher/MyAppDelegate.m +++ b/Mac/PythonLauncher/MyAppDelegate.m @@ -22,7 +22,7 @@ - (void)applicationDidFinishLaunching:(NSNotification *)notification { // Test that the file mappings are correct [self testFileTypeBinding]; - // If we were opened because of a file drag or doubleclick + // If we were opened because of a file drag or double-click // we've set initial_action_done in shouldShowUI // Otherwise we open a preferences dialog. if (!initial_action_done) { diff --git a/Mac/README.rst b/Mac/README.rst index 3ef2811da434c..8f2f153d50019 100644 --- a/Mac/README.rst +++ b/Mac/README.rst @@ -296,7 +296,7 @@ How do I create a binary distribution? Download and unpack the source release from https://www.python.org/download/. Go to the directory ``Mac/BuildScript``. There you will find a script ``build-installer.py`` that does all the work. This will download and build -a number of 3rd-party libaries, configures and builds a framework Python, +a number of 3rd-party libraries, configures and builds a framework Python, installs it, creates the installer package files and then packs this in a DMG image. The script also builds an HTML copy of the current Python documentation set for this release for inclusion in the framework. The diff --git a/Misc/NEWS.d/3.10.0a1.rst b/Misc/NEWS.d/3.10.0a1.rst index 044bd20594cc3..1c1c2d54e8c20 100644 --- a/Misc/NEWS.d/3.10.0a1.rst +++ b/Misc/NEWS.d/3.10.0a1.rst @@ -178,7 +178,7 @@ Convert the :mod:`_sha256` extension module types to heap types. .. section: Core and Builtins Fix a possible stack overflow in the parser when parsing functions and -classes with a huge ammount of arguments. Patch by Pablo Galindo. +classes with a huge amount of arguments. Patch by Pablo Galindo. .. diff --git a/Misc/NEWS.d/3.10.0a2.rst b/Misc/NEWS.d/3.10.0a2.rst index 3cfef17160336..61a291914f933 100644 --- a/Misc/NEWS.d/3.10.0a2.rst +++ b/Misc/NEWS.d/3.10.0a2.rst @@ -258,7 +258,7 @@ The :func:`repr` of :mod:`typing` types containing :ref:`Generic Alias Types .. nonce: 6aDbty .. section: Library -``binhex.binhex()`` consisently writes macOS 9 line endings. +``binhex.binhex()`` consistently writes macOS 9 line endings. .. @@ -534,7 +534,7 @@ belong to. Patch by Batuhan Taskaya. .. nonce: LR4fnY .. section: Library -Handle exceptions caused by unparseable date headers when using email +Handle exceptions caused by unparsable date headers when using email "default" policy. Patch by Tim Bell, Georges Toth .. diff --git a/Misc/NEWS.d/3.10.0a3.rst b/Misc/NEWS.d/3.10.0a3.rst index 0b76367f94445..4f182e8e3f1f0 100644 --- a/Misc/NEWS.d/3.10.0a3.rst +++ b/Misc/NEWS.d/3.10.0a3.rst @@ -949,7 +949,7 @@ branches, are now handled by docsbuild-script. .. nonce: W_updK .. section: Tests -Reenable test_gdb on gdb 9.2 and newer: +Re-enable test_gdb on gdb 9.2 and newer: https://bugzilla.redhat.com/show_bug.cgi?id=1866884 bug is fixed in gdb 10.1. diff --git a/Misc/NEWS.d/3.10.0a4.rst b/Misc/NEWS.d/3.10.0a4.rst index 57da9254587b4..214f9e521a55d 100644 --- a/Misc/NEWS.d/3.10.0a4.rst +++ b/Misc/NEWS.d/3.10.0a4.rst @@ -767,7 +767,7 @@ results. Patch by Ammar Askar. .. nonce: -7-XGz .. section: Tests -Update test_nntplib to use offical group name of news.aioe.org for testing. +Update test_nntplib to use official group name of news.aioe.org for testing. Patch by Dong-hee Na. .. @@ -863,7 +863,7 @@ and Peixing Xin. Now all platforms use a value for the "EXT_SUFFIX" build variable derived from SOABI (for instance in freeBSD, "EXT_SUFFIX" is now ".cpython-310d.so" -instead of ".so"). Previosuly only Linux, Mac and VxWorks were using a value +instead of ".so"). Previously only Linux, Mac and VxWorks were using a value for "EXT_SUFFIX" that included "SOABI". .. diff --git a/Misc/NEWS.d/3.10.0a5.rst b/Misc/NEWS.d/3.10.0a5.rst index 989edb0b47e97..1c7c7447cae06 100644 --- a/Misc/NEWS.d/3.10.0a5.rst +++ b/Misc/NEWS.d/3.10.0a5.rst @@ -549,7 +549,7 @@ Pass ``--timeout=$(TESTTIMEOUT)`` option to the default profile task .. section: Build Removed the grep -q and -E flags in the tzpath validation section of the -configure script to better accomodate users of some platforms (specifically +configure script to better accommodate users of some platforms (specifically Solaris 10). .. diff --git a/Misc/NEWS.d/3.10.0a6.rst b/Misc/NEWS.d/3.10.0a6.rst index 46d06add9115f..a4ee9ae098bd9 100644 --- a/Misc/NEWS.d/3.10.0a6.rst +++ b/Misc/NEWS.d/3.10.0a6.rst @@ -49,7 +49,7 @@ The :data:`types.FunctionType` constructor now inherits the current builtins if the *globals* dictionary has no ``"__builtins__"`` key, rather than using ``{"None": None}`` as builtins: same behavior as :func:`eval` and :func:`exec` functions. Defining a function with ``def function(...): ...`` -in Python is not affected, globals cannot be overriden with this syntax: it +in Python is not affected, globals cannot be overridden with this syntax: it also inherits the current builtins. Patch by Victor Stinner. .. @@ -355,7 +355,7 @@ in 4.0" to "3.12". See :pep:`623` for detail. .. nonce: LfTvL- .. section: Tests -Fix test_importlib to correctly skip Unicode file tests if the fileystem +Fix test_importlib to correctly skip Unicode file tests if the filesystem does not support them. .. diff --git a/Misc/NEWS.d/3.10.0a7.rst b/Misc/NEWS.d/3.10.0a7.rst index a20072b7ae11e..6c32e60dc8a46 100644 --- a/Misc/NEWS.d/3.10.0a7.rst +++ b/Misc/NEWS.d/3.10.0a7.rst @@ -195,7 +195,7 @@ decoded as Unicode characters outside the [U+0000; U+10ffff] range. .. nonce: lCzIg0 .. section: Core and Builtins -Fix a bug that was causing the parser to crash when emiting syntax errors +Fix a bug that was causing the parser to crash when emitting syntax errors when reading input from stdin. Patch by Pablo Galindo .. diff --git a/Misc/NEWS.d/3.10.0b1.rst b/Misc/NEWS.d/3.10.0b1.rst index a0dac135a3a60..ca524fe16b7cf 100644 --- a/Misc/NEWS.d/3.10.0b1.rst +++ b/Misc/NEWS.d/3.10.0b1.rst @@ -276,7 +276,7 @@ cause any runtime effects with ``from __future__ import annotations``. .. nonce: 0Ik1AM .. section: Core and Builtins -:exc:`SyntaxError` exceptions raised by the intepreter will highlight the +:exc:`SyntaxError` exceptions raised by the interpreter will highlight the full error range of the expression that consistutes the syntax error itself, instead of just where the problem is detected. Patch by Pablo Galindo. @@ -547,7 +547,7 @@ enum or one of its members' value. .. nonce: ejjsyR .. section: Library -For backwards compatbility with previous minor versions of Python, if +For backwards compatibility with previous minor versions of Python, if :func:`typing.get_type_hints` receives no namespace dictionary arguments, :func:`typing.get_type_hints` will search through the global then local namespaces during evaluation of stringized type annotations (string forward @@ -720,9 +720,9 @@ now raise ``TypeError`` during substitution. .. nonce: xT9QjF .. section: Library -The :mod:`multiprocessing` ``Server`` class now explicitly catchs +The :mod:`multiprocessing` ``Server`` class now explicitly catches :exc:`SystemExit` and closes the client connection in this case. It happens -when the ``Server.serve_client()`` method reachs the end of file (EOF). +when the ``Server.serve_client()`` method reaches the end of file (EOF). .. @@ -1132,7 +1132,7 @@ preferred "user", "home", and "prefix" (default) scheme names. .. section: Library Improve :meth:`sqlite3.Connection.backup` error handling. The error message -for non-existant target database names is now ``unknown database `` instead of ``SQL logic error``. Patch by Erlend E. Aasland. .. diff --git a/Misc/NEWS.d/3.9.0a1.rst b/Misc/NEWS.d/3.9.0a1.rst index 0a6a6eb287145..a9b6694c133f1 100644 --- a/Misc/NEWS.d/3.9.0a1.rst +++ b/Misc/NEWS.d/3.9.0a1.rst @@ -279,7 +279,7 @@ visited by ``tp_traverse()`` are valid. .. nonce: Xgc6F_ .. section: Core and Builtins -Remove unecessary intersection and update set operation in dictview with +Remove unnecessary intersection and update set operation in dictview with empty set. (Contributed by Dong-hee Na in :issue:`38210`.) .. @@ -868,7 +868,7 @@ Fix the :c:func:`PySys_Audit` call in :class:`mmap.mmap`. .. nonce: WJkgKV .. section: Core and Builtins -Remove an unnecssary Py_XINCREF in classobject.c. +Remove an unnecessary Py_XINCREF in classobject.c. .. @@ -1224,7 +1224,7 @@ Anthony Sottile. .. nonce: cyq5nr .. section: Library -Now :func:`~logging.config.fileConfig` correcty sets the .name of handlers +Now :func:`~logging.config.fileConfig` correctly sets the .name of handlers loaded. .. @@ -1637,7 +1637,7 @@ Preserve subclassing in inspect.Signature.from_callable. .. nonce: nHAbuJ .. section: Library -Names of hashing algorithms frome OpenSSL are now normalized to follow +Names of hashing algorithms from OpenSSL are now normalized to follow Python's naming conventions. For example OpenSSL uses sha3-512 instead of sha3_512 or blake2b512 instead of blake2b. @@ -2403,7 +2403,7 @@ Fixed comparisons of :class:`datetime.timedelta` and .. nonce: 7UV5d0 .. section: Library -Syncronize ``importlib.metadata`` with `importlib_metadata 0.19 +Synchronize ``importlib.metadata`` with `importlib_metadata 0.19 `_, improving handling of EGG-INFO files and fixing a crash when entry point names contained colons. @@ -3002,7 +3002,7 @@ on platforms with 16-bit :c:type:`wchar_t` (for example, Windows and AIX). In a subinterpreter, spawning a daemon thread now raises an exception. Daemon threads were never supported in subinterpreters. Previously, the -subinterpreter finalization crashed with a Pyton fatal error if a daemon +subinterpreter finalization crashed with a Python fatal error if a daemon thread was still running. .. @@ -3065,7 +3065,7 @@ internal tasks weak set is changed by another thread during iteration. .. section: Library :class:`_pyio.IOBase` destructor now does nothing if getting the ``closed`` -attribute fails to better mimick :class:`_io.IOBase` finalizer. +attribute fails to better mimic :class:`_io.IOBase` finalizer. .. @@ -4038,7 +4038,7 @@ crypto policies. Use PKCS#8 format with AES256 encryption instead. .. nonce: _x-9uH .. section: Tests -test.support now has a helper function to check for availibility of a hash +test.support now has a helper function to check for availability of a hash digest function. Several tests are refactored avoid MD5 and use SHA256 instead. Other tests are marked to use MD5 and skipped when MD5 is disabled. diff --git a/Misc/NEWS.d/3.9.0a3.rst b/Misc/NEWS.d/3.9.0a3.rst index 6c71d7e839d05..77ccc7453c215 100644 --- a/Misc/NEWS.d/3.9.0a3.rst +++ b/Misc/NEWS.d/3.9.0a3.rst @@ -178,7 +178,7 @@ last iteration of asynchronous for loops. Patch by Pablo Galindo. .. nonce: WG9alt .. section: Core and Builtins -Fix incorrent line execution reporting in trace functions when tracing +Fix incorrect line execution reporting in trace functions when tracing exception handlers with name binding. Patch by Pablo Galindo. .. @@ -685,7 +685,7 @@ but not required. Patch by Juergen Gmach. .. section: Library Fixed __subclasshook__ of :class:`os.PathLike` to return a correct result -upon inheritence. Patch by Bar Harel. +upon inheritance. Patch by Bar Harel. .. diff --git a/Misc/NEWS.d/3.9.0a5.rst b/Misc/NEWS.d/3.9.0a5.rst index 355a3fc22350c..49a118ad7e430 100644 --- a/Misc/NEWS.d/3.9.0a5.rst +++ b/Misc/NEWS.d/3.9.0a5.rst @@ -175,7 +175,7 @@ convention. Patch by Dong-hee Na. .. nonce: FE9S21 .. section: Core and Builtins -Chaged list overallocation strategy. It no longer overallocates if the new +Changed list overallocation strategy. It no longer overallocates if the new size is closer to overallocated size than to the old size and adds padding. .. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 76209100ecd2e..4a2ee18e8e975 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -285,7 +285,7 @@ macro Py_UNBLOCK_THREADS macro Py_END_ALLOW_THREADS added 3.2 -# The following were added in PC/python3.def in the intial stable ABI commit, +# The following were added in PC/python3.def in the initial stable ABI commit, # 4d0d471a8031de90a2b1ce99c4ac4780e60b3bc9, # and later amendments in 3.2: # 0d012f284be829c6217f60523db0e1671b7db9d9 From webhook-mailer at python.org Thu Oct 7 11:30:17 2021 From: webhook-mailer at python.org (JulienPalard) Date: Thu, 07 Oct 2021 15:30:17 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Python directory (GH-28767) (GH-28799) Message-ID: https://github.com/python/cpython/commit/ccd82a080056f21da2041d4c5d0ac4421ad0c8a8 commit: ccd82a080056f21da2041d4c5d0ac4421ad0c8a8 branch: 3.10 author: Christian Clauss committer: JulienPalard date: 2021-10-07T17:30:08+02:00 summary: [3.10] Fix typos in the Python directory (GH-28767) (GH-28799) (cherry picked from commit db693df3e112c5a61f2cbef63eedce3a36520ded) files: M Python/compile.c M Python/fileutils.c M Python/import.c M Python/initconfig.c M Python/pathconfig.c M Python/pythonrun.c diff --git a/Python/compile.c b/Python/compile.c index 97aa2247f6095..90861038e07da 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -6157,7 +6157,7 @@ compiler_pattern_or(struct compiler *c, pattern_ty p, pattern_context *pc) // cases, though. assert(istores < icontrol); Py_ssize_t rotations = istores + 1; - // Perfom the same rotation on pc->stores: + // Perform the same rotation on pc->stores: PyObject *rotated = PyList_GetSlice(pc->stores, 0, rotations); if (rotated == NULL || @@ -6785,7 +6785,7 @@ consts_dict_keys_inorder(PyObject *dict) return NULL; while (PyDict_Next(dict, &pos, &k, &v)) { i = PyLong_AS_LONG(v); - /* The keys of the dictionary can be tuples wrapping a contant. + /* The keys of the dictionary can be tuples wrapping a constant. * (see compiler_add_o and _PyCode_ConstantKey). In that case * the object we want is always second. */ if (PyTuple_CheckExact(k)) { diff --git a/Python/fileutils.c b/Python/fileutils.c index 9e732ddca55ce..c3144ee40782e 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -220,7 +220,7 @@ check_force_ascii(void) ch = (unsigned char)0xA7; res = _Py_mbstowcs(&wch, (char*)&ch, 1); if (res != DECODE_ERROR && wch == L'\xA7') { - /* On HP-UX withe C locale or the POSIX locale, + /* On HP-UX with C locale or the POSIX locale, nl_langinfo(CODESET) announces "roman8", whereas mbstowcs() uses Latin1 encoding in practice. Force ASCII in this case. diff --git a/Python/import.c b/Python/import.c index 50f4956e5a9d6..acfe96963ca34 100644 --- a/Python/import.c +++ b/Python/import.c @@ -2195,7 +2195,7 @@ _PyImport_BootstrapImp(PyThreadState *tstate) // Mock a ModuleSpec object just good enough for PyModule_FromDefAndSpec(): // an object with just a name attribute. // - // _imp.__spec__ is overriden by importlib._bootstrap._instal() anyway. + // _imp.__spec__ is overridden by importlib._bootstrap._instal() anyway. PyObject *attrs = Py_BuildValue("{sO}", "name", name); if (attrs == NULL) { goto error; diff --git a/Python/initconfig.c b/Python/initconfig.c index 27ae48dd3c97c..b2986116fdc35 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -2464,7 +2464,7 @@ warnoptions_append(PyConfig *config, PyWideStringList *options, { /* config_init_warnoptions() add existing config warnoptions at the end: ensure that the new option is not already present in this list to - prevent change the options order whne config_init_warnoptions() is + prevent change the options order when config_init_warnoptions() is called twice. */ if (_PyWideStringList_Find(&config->warnoptions, option)) { /* Already present: do nothing */ diff --git a/Python/pathconfig.c b/Python/pathconfig.c index 470aba75bea96..1017a571f2b82 100644 --- a/Python/pathconfig.c +++ b/Python/pathconfig.c @@ -405,7 +405,7 @@ config_init_pathconfig(PyConfig *config, int compute_path_config) #undef COPY_ATTR #ifdef MS_WINDOWS - /* If a ._pth file is found: isolated and site_import are overriden */ + /* If a ._pth file is found: isolated and site_import are overridden */ if (pathconfig.isolated != -1) { config->isolated = pathconfig.isolated; } diff --git a/Python/pythonrun.c b/Python/pythonrun.c index 8d9f6404fcad9..5704bcc7589f3 100644 --- a/Python/pythonrun.c +++ b/Python/pythonrun.c @@ -2,7 +2,7 @@ /* Top level execution of Python code (including in __main__) */ /* To help control the interfaces between the startup, execution and - * shutdown code, the phases are split across separate modules (boostrap, + * shutdown code, the phases are split across separate modules (bootstrap, * pythonrun, shutdown) */ @@ -943,7 +943,7 @@ print_exception(PyObject *f, PyObject *value) if (end_lineno > lineno) { end_offset = (error_line != NULL) ? line_size : -1; } - // Limit the ammount of '^' that we can display to + // Limit the amount of '^' that we can display to // the size of the text in the source line. if (error_line != NULL && end_offset > line_size + 1) { end_offset = line_size + 1; From webhook-mailer at python.org Thu Oct 7 11:31:29 2021 From: webhook-mailer at python.org (JulienPalard) Date: Thu, 07 Oct 2021 15:31:29 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Tools directory (GH-28769) (GH-28800) Message-ID: https://github.com/python/cpython/commit/da56601783312a74a5cd6ee8ea0881d534e0fc19 commit: da56601783312a74a5cd6ee8ea0881d534e0fc19 branch: 3.9 author: Christian Clauss committer: JulienPalard date: 2021-10-07T17:31:24+02:00 summary: [3.9] Fix typos in the Tools directory (GH-28769) (GH-28800) Like #28744 but for the Tools directory. Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 682aecfdeba481c876bfc9f3796c635bd5b5df50) files: M Tools/peg_generator/pegen/c_generator.py M Tools/peg_generator/pegen/first_sets.py M Tools/peg_generator/scripts/download_pypi_packages.py M Tools/pynche/ColorDB.py M Tools/scripts/dutree.doc diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index b4d6a0bab51f4..692ee69031ba5 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -685,7 +685,7 @@ def handle_alt_normal(self, node: Alt, is_gather: bool, rulename: Optional[str]) self.print( f'D(fprintf(stderr, "%*c+ {rulename}[%d-%d]: %s succeeded!\\n", p->level, \' \', _mark, p->mark, "{node_str}"));' ) - # Prepare to emmit the rule action and do so + # Prepare to emit the rule action and do so if node.action and "EXTRA" in node.action: self._set_up_token_end_metadata_extraction() if self.skip_actions: diff --git a/Tools/peg_generator/pegen/first_sets.py b/Tools/peg_generator/pegen/first_sets.py index 71be5a2e7cbf4..ce80bb5d5c446 100755 --- a/Tools/peg_generator/pegen/first_sets.py +++ b/Tools/peg_generator/pegen/first_sets.py @@ -59,7 +59,7 @@ def visit_Alt(self, item: Alt) -> Set[str]: result -= to_remove # If the set of new terminals can start with the empty string, - # it means that the item is completelly nullable and we should + # it means that the item is completely nullable and we should # also considering at least the next item in case the current # one fails to parse. diff --git a/Tools/peg_generator/scripts/download_pypi_packages.py b/Tools/peg_generator/scripts/download_pypi_packages.py index 9874202d379ee..ca1216087a1fd 100755 --- a/Tools/peg_generator/scripts/download_pypi_packages.py +++ b/Tools/peg_generator/scripts/download_pypi_packages.py @@ -72,7 +72,7 @@ def main() -> None: package_json = load_json(package_name) try: - print(f"Dowloading and compressing package {package_name} ... ", end="") + print(f"Downloading and compressing package {package_name} ... ", end="") download_package_code(package_name, package_json) print("Done") except (IndexError, KeyError): diff --git a/Tools/pynche/ColorDB.py b/Tools/pynche/ColorDB.py index eb76d4042d33b..c013a60896908 100644 --- a/Tools/pynche/ColorDB.py +++ b/Tools/pynche/ColorDB.py @@ -9,7 +9,7 @@ trouble reading the file, None is returned. You can pass get_colordb() an optional filetype argument. -Supporte file types are: +Supported file types are: X_RGB_TXT -- X Consortium rgb.txt format files. Three columns of numbers from 0 .. 255 separated by whitespace. Arbitrary trailing diff --git a/Tools/scripts/dutree.doc b/Tools/scripts/dutree.doc index 97bd2e2e47cae..490126b0182d1 100644 --- a/Tools/scripts/dutree.doc +++ b/Tools/scripts/dutree.doc @@ -15,7 +15,7 @@ From the keyboard of flee at cs.psu.edu (Felix Lee): :And Perl is definitely awkward with data types. I haven't yet found a :pleasant way of shoving non-trivial data types into Perl's grammar. -Yes, it's pretty aweful at that, alright. Sometimes I write perl programs +Yes, it's pretty awful at that, alright. Sometimes I write perl programs that need them, and sometimes it just takes a little creativity. But sometimes it's not worth it. I actually wrote a C program the other day (gasp) because I didn't want to deal with a game matrix with six links per node. From webhook-mailer at python.org Thu Oct 7 11:35:08 2021 From: webhook-mailer at python.org (JulienPalard) Date: Thu, 07 Oct 2021 15:35:08 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Tools directory (GH-28769) (GH-28801) Message-ID: https://github.com/python/cpython/commit/03bf55d8cf563ccdee2ec61c5c734c59b5d94ba4 commit: 03bf55d8cf563ccdee2ec61c5c734c59b5d94ba4 branch: 3.10 author: Christian Clauss committer: JulienPalard date: 2021-10-07T17:34:55+02:00 summary: [3.10] Fix typos in the Tools directory (GH-28769) (GH-28801) Like GH-28744 but for the Tools directory. Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 682aecfdeba481c876bfc9f3796c635bd5b5df50) Co-authored-by: Christian Clauss files: M Tools/c-analyzer/c_analyzer/__main__.py M Tools/c-analyzer/c_analyzer/info.py M Tools/c-analyzer/c_common/logging.py M Tools/c-analyzer/c_common/strutil.py M Tools/c-analyzer/c_parser/info.py M Tools/c-analyzer/c_parser/parser/__init__.py M Tools/c-analyzer/c_parser/preprocessor/__init__.py M Tools/c-analyzer/cpython/__main__.py M Tools/c-analyzer/cpython/ignored.tsv M Tools/peg_generator/pegen/c_generator.py M Tools/peg_generator/pegen/first_sets.py M Tools/peg_generator/scripts/download_pypi_packages.py M Tools/pynche/ColorDB.py M Tools/scripts/dutree.doc M Tools/scripts/stable_abi.py diff --git a/Tools/c-analyzer/c_analyzer/__main__.py b/Tools/c-analyzer/c_analyzer/__main__.py index 24fc6cd182656..5d89b29adf899 100644 --- a/Tools/c-analyzer/c_analyzer/__main__.py +++ b/Tools/c-analyzer/c_analyzer/__main__.py @@ -482,7 +482,7 @@ def cmd_data(datacmd, filenames, known=None, *, cmd_analyze, ), 'data': ( - 'check/manage local data (e.g. knwon types, ignored vars, caches)', + 'check/manage local data (e.g. known types, ignored vars, caches)', [_cli_data], cmd_data, ), diff --git a/Tools/c-analyzer/c_analyzer/info.py b/Tools/c-analyzer/c_analyzer/info.py index b75918e5e7a68..27c3a5a4ee76f 100644 --- a/Tools/c-analyzer/c_analyzer/info.py +++ b/Tools/c-analyzer/c_analyzer/info.py @@ -230,11 +230,11 @@ def fix_filename(self, relroot=fsutil.USE_CWD, **kwargs): return self def as_rowdata(self, columns=None): - # XXX finsih! + # XXX finish! return self.item.as_rowdata(columns) def render_rowdata(self, columns=None): - # XXX finsih! + # XXX finish! return self.item.render_rowdata(columns) def render(self, fmt='line', *, itemonly=False): diff --git a/Tools/c-analyzer/c_common/logging.py b/Tools/c-analyzer/c_common/logging.py index 12398f7e385fd..10af852ec3c5e 100644 --- a/Tools/c-analyzer/c_common/logging.py +++ b/Tools/c-analyzer/c_common/logging.py @@ -41,7 +41,7 @@ def configure_logger(logger, verbosity=VERBOSITY, *, def hide_emit_errors(): """Ignore errors while emitting log entries. - Rather than printing a message desribing the error, we show nothing. + Rather than printing a message describing the error, we show nothing. """ # For now we simply ignore all exceptions. If we wanted to ignore # specific ones (e.g. BrokenPipeError) then we would need to use diff --git a/Tools/c-analyzer/c_common/strutil.py b/Tools/c-analyzer/c_common/strutil.py index e7535d45bbba2..07193c091e4c3 100644 --- a/Tools/c-analyzer/c_common/strutil.py +++ b/Tools/c-analyzer/c_common/strutil.py @@ -26,7 +26,7 @@ def parse_entries(entries, *, ignoresep=None): # We read the entire file here to ensure the file # gets closed sooner rather than later. Note that # the file would stay open if this iterator is never - # exchausted. + # exhausted. lines = infile.read().splitlines() for line in _iter_significant_lines(lines): yield line, filename diff --git a/Tools/c-analyzer/c_parser/info.py b/Tools/c-analyzer/c_parser/info.py index 98ff511cfe64a..697b1f26dc215 100644 --- a/Tools/c-analyzer/c_parser/info.py +++ b/Tools/c-analyzer/c_parser/info.py @@ -1029,7 +1029,7 @@ def _resolve_data(cls, data): @classmethod def _raw_data(self, data): - # XXX finsh! + # XXX finish! return data @classmethod @@ -1255,7 +1255,7 @@ def _resolve_data(cls, data): @classmethod def _raw_data(self, data): - # XXX finsih! + # XXX finish! return data @classmethod @@ -1296,12 +1296,12 @@ class Statement(HighlevelParsedItem): @classmethod def _resolve_data(cls, data): - # XXX finsih! + # XXX finish! return data, None @classmethod def _raw_data(self, data): - # XXX finsih! + # XXX finish! return data @classmethod diff --git a/Tools/c-analyzer/c_parser/parser/__init__.py b/Tools/c-analyzer/c_parser/parser/__init__.py index 39056099f5e91..df70aae66b776 100644 --- a/Tools/c-analyzer/c_parser/parser/__init__.py +++ b/Tools/c-analyzer/c_parser/parser/__init__.py @@ -7,7 +7,7 @@ Furthermore, the grammar rules for the C syntax (particularly as described in the K&R book) actually describe a superset, of which the -full C langage is a proper subset. Here are some of the extra +full C language is a proper subset. Here are some of the extra conditions that must be applied when parsing C code: * ... @@ -90,7 +90,7 @@ * no "inline" type declarations (struct, union, enum) in function parameters ~(including function pointers)~ * no "inline" type decls in function return types -* no superflous parentheses in declarators +* no superfluous parentheses in declarators * var decls in for loops are always "simple" (e.g. no inline types) * only inline struct/union/enum decls may be anonymouns (without a name) * no function pointers in function pointer parameters diff --git a/Tools/c-analyzer/c_parser/preprocessor/__init__.py b/Tools/c-analyzer/c_parser/preprocessor/__init__.py index 8da4d8cadf7d0..e38176fee31fa 100644 --- a/Tools/c-analyzer/c_parser/preprocessor/__init__.py +++ b/Tools/c-analyzer/c_parser/preprocessor/__init__.py @@ -19,7 +19,7 @@ logger = logging.getLogger(__name__) -# Supprted "source": +# Supported "source": # * filename (string) # * lines (iterable) # * text (string) @@ -156,7 +156,7 @@ def handling_errors(ignore_exc=None, *, log_err=None): # tools _COMPILERS = { - # matching disutils.ccompiler.compiler_class: + # matching distutils.ccompiler.compiler_class: 'unix': _gcc.preprocess, 'msvc': None, 'cygwin': None, diff --git a/Tools/c-analyzer/cpython/__main__.py b/Tools/c-analyzer/cpython/__main__.py index a11b687214d2f..06ec871ba75e3 100644 --- a/Tools/c-analyzer/cpython/__main__.py +++ b/Tools/c-analyzer/cpython/__main__.py @@ -342,7 +342,7 @@ def cmd_capi(filenames=None, *, cmd_parse, ), 'data': ( - 'check/manage local data (e.g. knwon types, ignored vars, caches)', + 'check/manage local data (e.g. known types, ignored vars, caches)', [_cli_data], cmd_data, ), diff --git a/Tools/c-analyzer/cpython/ignored.tsv b/Tools/c-analyzer/cpython/ignored.tsv index e5d93782076c3..91867cae8d1e0 100644 --- a/Tools/c-analyzer/cpython/ignored.tsv +++ b/Tools/c-analyzer/cpython/ignored.tsv @@ -2128,7 +2128,7 @@ Python/import.c import_find_and_load header - #----------------------- # runtime state -# (look at the bottome of the file) +# (look at the bottom of the file) #----------------------- # modules diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index c1ab5e0ded178..7941978252d17 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -723,7 +723,7 @@ def handle_alt_normal(self, node: Alt, is_gather: bool, rulename: Optional[str]) self.print( f'D(fprintf(stderr, "%*c+ {rulename}[%d-%d]: %s succeeded!\\n", p->level, \' \', _mark, p->mark, "{node_str}"));' ) - # Prepare to emmit the rule action and do so + # Prepare to emit the rule action and do so if node.action and "EXTRA" in node.action: self._set_up_token_end_metadata_extraction() if self.skip_actions: diff --git a/Tools/peg_generator/pegen/first_sets.py b/Tools/peg_generator/pegen/first_sets.py index 71be5a2e7cbf4..ce80bb5d5c446 100755 --- a/Tools/peg_generator/pegen/first_sets.py +++ b/Tools/peg_generator/pegen/first_sets.py @@ -59,7 +59,7 @@ def visit_Alt(self, item: Alt) -> Set[str]: result -= to_remove # If the set of new terminals can start with the empty string, - # it means that the item is completelly nullable and we should + # it means that the item is completely nullable and we should # also considering at least the next item in case the current # one fails to parse. diff --git a/Tools/peg_generator/scripts/download_pypi_packages.py b/Tools/peg_generator/scripts/download_pypi_packages.py index 9874202d379ee..ca1216087a1fd 100755 --- a/Tools/peg_generator/scripts/download_pypi_packages.py +++ b/Tools/peg_generator/scripts/download_pypi_packages.py @@ -72,7 +72,7 @@ def main() -> None: package_json = load_json(package_name) try: - print(f"Dowloading and compressing package {package_name} ... ", end="") + print(f"Downloading and compressing package {package_name} ... ", end="") download_package_code(package_name, package_json) print("Done") except (IndexError, KeyError): diff --git a/Tools/pynche/ColorDB.py b/Tools/pynche/ColorDB.py index eb76d4042d33b..c013a60896908 100644 --- a/Tools/pynche/ColorDB.py +++ b/Tools/pynche/ColorDB.py @@ -9,7 +9,7 @@ trouble reading the file, None is returned. You can pass get_colordb() an optional filetype argument. -Supporte file types are: +Supported file types are: X_RGB_TXT -- X Consortium rgb.txt format files. Three columns of numbers from 0 .. 255 separated by whitespace. Arbitrary trailing diff --git a/Tools/scripts/dutree.doc b/Tools/scripts/dutree.doc index 97bd2e2e47cae..490126b0182d1 100644 --- a/Tools/scripts/dutree.doc +++ b/Tools/scripts/dutree.doc @@ -15,7 +15,7 @@ From the keyboard of flee at cs.psu.edu (Felix Lee): :And Perl is definitely awkward with data types. I haven't yet found a :pleasant way of shoving non-trivial data types into Perl's grammar. -Yes, it's pretty aweful at that, alright. Sometimes I write perl programs +Yes, it's pretty awful at that, alright. Sometimes I write perl programs that need them, and sometimes it just takes a little creativity. But sometimes it's not worth it. I actually wrote a C program the other day (gasp) because I didn't want to deal with a game matrix with six links per node. diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index b7fd2c8583ba7..6d7034090f881 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -67,7 +67,7 @@ class Manifest: def add(self, item): if item.name in self.contents: # We assume that stable ABI items do not share names, - # even if they're diferent kinds (e.g. function vs. macro). + # even if they're different kinds (e.g. function vs. macro). raise ValueError(f'duplicate ABI item {item.name}') self.contents[item.name] = item @@ -295,7 +295,7 @@ def do_unixy_check(manifest, args): present_macros = gcc_get_limited_api_macros(['Include/Python.h']) feature_defines = manifest.feature_defines & present_macros - # Check that we have all neded macros + # Check that we have all needed macros expected_macros = set( item.name for item in manifest.select({'macro'}) ) @@ -412,7 +412,7 @@ def binutils_check_library(manifest, library, expected_symbols, dynamic): def gcc_get_limited_api_macros(headers): """Get all limited API macros from headers. - Runs the preprocesor over all the header files in "Include" setting + Runs the preprocessor over all the header files in "Include" setting "-DPy_LIMITED_API" to the correct value for the running version of the interpreter and extracting all macro definitions (via adding -dM to the compiler arguments). @@ -449,7 +449,7 @@ def gcc_get_limited_api_macros(headers): def gcc_get_limited_api_definitions(headers): """Get all limited API definitions from headers. - Run the preprocesor over all the header files in "Include" setting + Run the preprocessor over all the header files in "Include" setting "-DPy_LIMITED_API" to the correct value for the running version of the interpreter. From webhook-mailer at python.org Thu Oct 7 11:42:48 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 15:42:48 -0000 Subject: [Python-checkins] [3.9] Fix typos in the Lib directory (GH-28775) (GH-28803) Message-ID: https://github.com/python/cpython/commit/4346b81cc60ee23e0ae175300822615bd7d16431 commit: 4346b81cc60ee23e0ae175300822615bd7d16431 branch: 3.9 author: Christian Clauss committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T08:42:38-07:00 summary: [3.9] Fix typos in the Lib directory (GH-28775) (GH-28803) Fix typos in the Lib directory as identified by codespell. Co-authored-by: Terry Jan Reedy (cherry picked from commit 745c9d9dfc1ad6fdfdf1d07420c6273ff67fa5be) Automerge-Triggered-By: GH:JulienPalard files: M Lib/asyncio/events.py M Lib/asyncio/unix_events.py M Lib/ctypes/_aix.py M Lib/ctypes/test/test_structures.py M Lib/difflib.py M Lib/distutils/ccompiler.py M Lib/html/parser.py M Lib/idlelib/ChangeLog M Lib/idlelib/idle_test/htest.py M Lib/idlelib/idle_test/mock_tk.py M Lib/idlelib/idle_test/test_multicall.py M Lib/idlelib/idle_test/test_pyparse.py M Lib/lib2to3/fixes/fix_metaclass.py M Lib/lib2to3/fixes/fix_paren.py M Lib/linecache.py M Lib/pickle.py M Lib/sqlite3/test/dbapi.py M Lib/test/datetimetester.py M Lib/test/decimaltestdata/abs.decTest M Lib/test/decimaltestdata/extra.decTest M Lib/test/libregrtest/refleak.py M Lib/test/pickletester.py M Lib/test/test__xxsubinterpreters.py M Lib/test/test_asyncio/test_streams.py M Lib/test/test_asyncio/test_subprocess.py M Lib/test/test_asyncio/test_tasks.py M Lib/test/test_collections.py M Lib/test/test_descr.py M Lib/test/test_dict.py M Lib/test/test_dict_version.py M Lib/test/test_dtrace.py M Lib/test/test_email/test_message.py M Lib/test/test_embed.py M Lib/test/test_future.py M Lib/test/test_lltrace.py M Lib/test/test_locale.py M Lib/test/test_pathlib.py M Lib/test/test_strftime.py M Lib/test/test_sys.py M Lib/test/test_time.py M Lib/test/test_unparse.py M Lib/test/test_weakref.py M Lib/test/test_wsgiref.py M Lib/test/test_xml_etree.py M Lib/test/test_xmlrpc.py M Lib/threading.py M Lib/tkinter/__init__.py M Lib/tkinter/test/test_ttk/test_widgets.py M Lib/unittest/async_case.py M Lib/unittest/test/testmock/testsealable.py M Lib/venv/scripts/common/Activate.ps1 M Lib/wsgiref/validate.py M Lib/zoneinfo/_zoneinfo.py diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index 0dce87b8ecc58..c8e4b407b476d 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -465,7 +465,7 @@ async def connect_read_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def connect_write_pipe(self, protocol_factory, pipe): @@ -478,7 +478,7 @@ async def connect_write_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def subprocess_shell(self, protocol_factory, cmd, *, diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 56fcc084e311f..9ab5dba272fd1 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1383,7 +1383,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classe requires it + # abstract base classes requires it return True def attach_loop(self, loop): diff --git a/Lib/ctypes/_aix.py b/Lib/ctypes/_aix.py index 26959d90a4dd6..fc3e95cbcc88a 100644 --- a/Lib/ctypes/_aix.py +++ b/Lib/ctypes/_aix.py @@ -163,7 +163,7 @@ def get_legacy(members): return member else: # 32-bit legacy names - both shr.o and shr4.o exist. - # shr.o is the preffered name so we look for shr.o first + # shr.o is the preferred name so we look for shr.o first # i.e., shr4.o is returned only when shr.o does not exist for name in ['shr.o', 'shr4.o']: member = get_one_match(re.escape(name), members) diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py index 245cd94c5cdd9..97ad2b8ed8a50 100644 --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -443,7 +443,7 @@ def __del__(self): s = Test(1, 2, 3) # Test the StructUnionType_paramfunc() code path which copies the - # structure: if the stucture is larger than sizeof(void*). + # structure: if the structure is larger than sizeof(void*). self.assertGreater(sizeof(s), sizeof(c_void_p)) dll = CDLL(_ctypes_test.__file__) @@ -451,7 +451,7 @@ def __del__(self): func.argtypes = (Test,) func.restype = None func(s) - # bpo-37140: Passing the structure by refrence must not call + # bpo-37140: Passing the structure by reference must not call # its finalizer! self.assertEqual(finalizer_calls, []) self.assertEqual(s.first, 1) diff --git a/Lib/difflib.py b/Lib/difflib.py index 480bad2224c8e..afd8a0c7c5b61 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -62,7 +62,7 @@ class SequenceMatcher: notion, pairing up elements that appear uniquely in each sequence. That, and the method here, appear to yield more intuitive difference reports than does diff. This method appears to be the least vulnerable - to synching up on blocks of "junk lines", though (like blank lines in + to syncing up on blocks of "junk lines", though (like blank lines in ordinary text files, or maybe "

" lines in HTML files). That may be because this is the only method of the 3 that has a *concept* of "junk" . diff --git a/Lib/distutils/ccompiler.py b/Lib/distutils/ccompiler.py index b5ef143e72c56..4c47f2ed245d4 100644 --- a/Lib/distutils/ccompiler.py +++ b/Lib/distutils/ccompiler.py @@ -392,7 +392,7 @@ def _fix_compile_args(self, output_dir, macros, include_dirs): return output_dir, macros, include_dirs def _prep_compile(self, sources, output_dir, depends=None): - """Decide which souce files must be recompiled. + """Decide which source files must be recompiled. Determine the list of object files corresponding to 'sources', and figure out which ones really need to be recompiled. diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 9e49effca1fcc..58f6bb3b1e932 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -405,7 +405,7 @@ def parse_endtag(self, i): tagname = namematch.group(1).lower() # consume and ignore other stuff between the name and the > # Note: this is not 100% correct, since we might have things like - # , but looking for > after tha name should cover + # , but looking for > after the name should cover # most of the cases and is much simpler gtpos = rawdata.find('>', namematch.end()) self.handle_endtag(tagname) diff --git a/Lib/idlelib/ChangeLog b/Lib/idlelib/ChangeLog index d7d7e1efdb1d3..c8960cfa535d0 100644 --- a/Lib/idlelib/ChangeLog +++ b/Lib/idlelib/ChangeLog @@ -1175,7 +1175,7 @@ Wed Mar 10 05:18:02 1999 Guido van Rossum classes in selected module methods of selected class - Sinlge clicking in a directory, module or class item updates the next + Single clicking in a directory, module or class item updates the next column with info about the selected item. Double clicking in a module, class or method item opens the file (and selects the clicked item if it is a class or method). diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py index 1373b7642a6ea..666ff4cb84851 100644 --- a/Lib/idlelib/idle_test/htest.py +++ b/Lib/idlelib/idle_test/htest.py @@ -246,7 +246,7 @@ def _wrapper(parent): # htest # _object_browser_spec = { 'file': 'debugobj', 'kwds': {}, - 'msg': "Double click on items upto the lowest level.\n" + 'msg': "Double click on items up to the lowest level.\n" "Attributes of the objects and related information " "will be displayed side-by-side at each level." } @@ -255,7 +255,7 @@ def _wrapper(parent): # htest # 'file': 'pathbrowser', 'kwds': {}, 'msg': "Test for correct display of all paths in sys.path.\n" - "Toggle nested items upto the lowest level.\n" + "Toggle nested items up to the lowest level.\n" "Double clicking on an item prints a traceback\n" "for an exception that is ignored." } @@ -341,7 +341,7 @@ def _wrapper(parent): # htest # 'file': 'tree', 'kwds': {}, 'msg': "The canvas is scrollable.\n" - "Click on folders upto to the lowest level." + "Click on folders up to to the lowest level." } _undo_delegator_spec = { diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py index db583553838fb..8304734b847a8 100644 --- a/Lib/idlelib/idle_test/mock_tk.py +++ b/Lib/idlelib/idle_test/mock_tk.py @@ -79,7 +79,7 @@ def tearDownClass(cls): --- For 'ask' functions, set func.result return value before calling the method that uses the message function. When messagebox functions are the - only gui alls in a method, this replacement makes the method gui-free, + only GUI calls in a method, this replacement makes the method GUI-free, """ askokcancel = Mbox_func() # True or False askquestion = Mbox_func() # 'yes' or 'no' diff --git a/Lib/idlelib/idle_test/test_multicall.py b/Lib/idlelib/idle_test/test_multicall.py index ba582bb3ca51b..b3a3bfb88f9c3 100644 --- a/Lib/idlelib/idle_test/test_multicall.py +++ b/Lib/idlelib/idle_test/test_multicall.py @@ -37,7 +37,7 @@ def test_init(self): def test_yview(self): # Added for tree.wheel_event - # (it depends on yview to not be overriden) + # (it depends on yview to not be overridden) mc = self.mc self.assertIs(mc.yview, Text.yview) mctext = self.mc(self.root) diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index fb5726db1d821..384db566ac76c 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -284,7 +284,7 @@ def test_get_num_lines_in_stmt(self): tests = ( TestInfo('[x for x in a]\n', 1), # Closed on one line. TestInfo('[x\nfor x in a\n', 2), # Not closed. - TestInfo('[x\\\nfor x in a\\\n', 2), # "", uneeded backslashes. + TestInfo('[x\\\nfor x in a\\\n', 2), # "", unneeded backslashes. TestInfo('[x\nfor x in a\n]\n', 3), # Closed on multi-line. TestInfo('\n"""Docstring comment L1"""\nL2\nL3\nL4\n', 1), TestInfo('\n"""Docstring comment L1\nL2"""\nL3\nL4\n', 1), diff --git a/Lib/lib2to3/fixes/fix_metaclass.py b/Lib/lib2to3/fixes/fix_metaclass.py index d1cd10d327587..fe547b2228072 100644 --- a/Lib/lib2to3/fixes/fix_metaclass.py +++ b/Lib/lib2to3/fixes/fix_metaclass.py @@ -51,7 +51,7 @@ def fixup_parse_tree(cls_node): # already in the preferred format, do nothing return - # !%@#! oneliners have no suite node, we have to fake one up + # !%@#! one-liners have no suite node, we have to fake one up for i, node in enumerate(cls_node.children): if node.type == token.COLON: break diff --git a/Lib/lib2to3/fixes/fix_paren.py b/Lib/lib2to3/fixes/fix_paren.py index b205aa7e1e93f..df3da5f5232c9 100644 --- a/Lib/lib2to3/fixes/fix_paren.py +++ b/Lib/lib2to3/fixes/fix_paren.py @@ -1,4 +1,4 @@ -"""Fixer that addes parentheses where they are required +"""Fixer that adds parentheses where they are required This converts ``[x for x in 1, 2]`` to ``[x for x in (1, 2)]``.""" diff --git a/Lib/linecache.py b/Lib/linecache.py index fa5dbd09eab86..921bfa039bd48 100644 --- a/Lib/linecache.py +++ b/Lib/linecache.py @@ -154,7 +154,7 @@ def lazycache(filename, module_globals): :return: True if a lazy load is registered in the cache, otherwise False. To register such a load a module loader with a - get_source method must be found, the filename must be a cachable + get_source method must be found, the filename must be a cacheable filename, and the filename must not be already cached. """ if filename in cache: diff --git a/Lib/pickle.py b/Lib/pickle.py index e63a8b6e4dbb7..3d2c75a853889 100644 --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -1172,7 +1172,7 @@ def __init__(self, file, *, fix_imports=True, used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python 2; these default to 'ASCII' and 'strict', respectively. *encoding* can be - 'bytes' to read theses 8-bit string instances as bytes objects. + 'bytes' to read these 8-bit string instances as bytes objects. """ self._buffers = iter(buffers) if buffers is not None else None self._file_readline = file.readline diff --git a/Lib/sqlite3/test/dbapi.py b/Lib/sqlite3/test/dbapi.py index d00b60cac4e38..053543c0536f9 100644 --- a/Lib/sqlite3/test/dbapi.py +++ b/Lib/sqlite3/test/dbapi.py @@ -1,7 +1,7 @@ #-*- coding: iso-8859-1 -*- # pysqlite2/test/dbapi.py: tests for DB-API compliance # -# Copyright (C) 2004-2010 Gerhard H?ring +# Copyright (C) 2004-2010 Gerhard H??ring # # This file is part of pysqlite. # @@ -451,7 +451,7 @@ def CheckFetchoneNoStatement(self): self.assertEqual(row, None) def CheckArraySize(self): - # must default ot 1 + # must default to 1 self.assertEqual(self.cu.arraysize, 1) # now set to 2 diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py index fcc13bfc59667..85241526739cb 100644 --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -4064,7 +4064,7 @@ def test_even_more_compare(self): self.assertEqual(t1, t1) self.assertEqual(t2, t2) - # Equal afer adjustment. + # Equal after adjustment. t1 = self.theclass(1, 12, 31, 23, 59, tzinfo=FixedOffset(1, "")) t2 = self.theclass(2, 1, 1, 3, 13, tzinfo=FixedOffset(3*60+13+2, "")) self.assertEqual(t1, t2) @@ -4903,7 +4903,7 @@ def test_easy(self): # OTOH, these fail! Don't enable them. The difficulty is that # the edge case tests assume that every hour is representable in # the "utc" class. This is always true for a fixed-offset tzinfo - # class (lke utc_real and utc_fake), but not for Eastern or Central. + # class (like utc_real and utc_fake), but not for Eastern or Central. # For these adjacent DST-aware time zones, the range of time offsets # tested ends up creating hours in the one that aren't representable # in the other. For the same reason, we would see failures in the diff --git a/Lib/test/decimaltestdata/abs.decTest b/Lib/test/decimaltestdata/abs.decTest index 01f73d7766648..569b8fcd84ab6 100644 --- a/Lib/test/decimaltestdata/abs.decTest +++ b/Lib/test/decimaltestdata/abs.decTest @@ -20,7 +20,7 @@ version: 2.59 -- This set of tests primarily tests the existence of the operator. --- Additon, subtraction, rounding, and more overflows are tested +-- Addition, subtraction, rounding, and more overflows are tested -- elsewhere. precision: 9 diff --git a/Lib/test/decimaltestdata/extra.decTest b/Lib/test/decimaltestdata/extra.decTest index b630d8e3f9d45..31291202a35e7 100644 --- a/Lib/test/decimaltestdata/extra.decTest +++ b/Lib/test/decimaltestdata/extra.decTest @@ -156,7 +156,7 @@ extr1302 fma -Inf 0E-456 sNaN148 -> NaN Invalid_operation -- max/min/max_mag/min_mag bug in 2.5.2/2.6/3.0: max(NaN, finite) gave -- incorrect answers when the finite number required rounding; similarly --- for the other thre functions +-- for the other three functions maxexponent: 999 minexponent: -999 precision: 6 diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 8d221232eb6ce..061204114d275 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -109,7 +109,7 @@ def get_pooled_int(value): # These checkers return False on success, True on failure def check_rc_deltas(deltas): - # Checker for reference counters and memomry blocks. + # Checker for reference counters and memory blocks. # # bpo-30776: Try to ignore false positives: # diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index a4ce5cebaebbb..e86847c5f13d8 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3711,7 +3711,7 @@ class MyClass: self.assertEqual(new_f, 5) self.assertEqual(some_str, 'some str') - # math.log does not have its usual reducer overriden, so the + # math.log does not have its usual reducer overridden, so the # custom reduction callback should silently direct the pickler # to the default pickling by attribute, by returning # NotImplemented @@ -3728,7 +3728,7 @@ class MyClass: def test_reducer_override_no_reference_cycle(self): # bpo-39492: reducer_override used to induce a spurious reference cycle # inside the Pickler object, that could prevent all serialized objects - # from being garbage-collected without explicity invoking gc.collect. + # from being garbage-collected without explicitly invoking gc.collect. for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): with self.subTest(proto=proto): diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py index cbfa881818bc6..6260d0937eec8 100644 --- a/Lib/test/test__xxsubinterpreters.py +++ b/Lib/test/test__xxsubinterpreters.py @@ -1231,7 +1231,7 @@ def test_channel_list_interpreters_basic(self): import _xxsubinterpreters as _interpreters obj = _interpreters.channel_recv({cid}) """)) - # Test for channel that has boths ends associated to an interpreter. + # Test for channel that has both ends associated to an interpreter. send_interps = interpreters.channel_list_interpreters(cid, send=True) recv_interps = interpreters.channel_list_interpreters(cid, send=False) self.assertEqual(send_interps, [interp0]) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 71de82dfc7ebb..38f4d84d4755f 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -736,7 +736,7 @@ def test_read_all_from_pipe_reader(self): # See asyncio issue 168. This test is derived from the example # subprocess_attach_read_pipe.py, but we configure the # StreamReader's limit so that twice it is less than the size - # of the data writter. Also we must explicitly attach a child + # of the data writer. Also we must explicitly attach a child # watcher to the event loop. code = """\ diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 693cc41290430..3f6fc15b278b8 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -227,7 +227,7 @@ def prepare_broken_pipe_test(self): # buffer large enough to feed the whole pipe buffer large_data = b'x' * support.PIPE_MAX_SIZE - # the program ends before the stdin can be feeded + # the program ends before the stdin can be fed proc = self.loop.run_until_complete( asyncio.create_subprocess_exec( sys.executable, '-c', 'pass', diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index e25c0a697fa18..402edaec0d8f9 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -3568,7 +3568,7 @@ def test_run_coroutine_threadsafe_with_timeout(self): self.assertTrue(task.done()) def test_run_coroutine_threadsafe_task_cancelled(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task is cancelled.""" callback = lambda: self.target(cancel=True) future = self.loop.run_in_executor(None, callback) @@ -3576,7 +3576,7 @@ def test_run_coroutine_threadsafe_task_cancelled(self): self.loop.run_until_complete(future) def test_run_coroutine_threadsafe_task_factory_exception(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task factory raise an exception.""" def task_factory(loop, coro): diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index 4ae9748e8eafe..9e1174743591c 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -1584,7 +1584,7 @@ def assertSameSet(self, s1, s2): self.assertSetEqual(set(s1), set(s2)) def test_Set_from_iterable(self): - """Verify _from_iterable overriden to an instance method works.""" + """Verify _from_iterable overridden to an instance method works.""" class SetUsingInstanceFromIterable(MutableSet): def __init__(self, values, created_by): if not created_by: diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index 2fa68b201172f..0e6a784c6c14b 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -5672,7 +5672,7 @@ class A(metaclass=M): def test_incomplete_super(self): """ - Attrubute lookup on a super object must be aware that + Attribute lookup on a super object must be aware that its target type can be uninitialized (type->tp_mro == NULL). """ class M(DebugHelperMeta): diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index b10c07534ddb2..d05ed302ede84 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1021,7 +1021,7 @@ def test_splittable_pop(self): @support.cpython_only def test_splittable_pop_pending(self): - """pop a pending key in a splitted table should not crash""" + """pop a pending key in a split table should not crash""" a, b = self.make_shared_key_dict(2) a['a'] = 4 @@ -1368,7 +1368,7 @@ def test_reversed(self): self.assertRaises(StopIteration, next, r) def test_reverse_iterator_for_empty_dict(self): - # bpo-38525: revered iterator should work properly + # bpo-38525: reversed iterator should work properly # empty dict is directly used for reference count test self.assertEqual(list(reversed({})), []) diff --git a/Lib/test/test_dict_version.py b/Lib/test/test_dict_version.py index b23786514f82e..95a42d03e6850 100644 --- a/Lib/test/test_dict_version.py +++ b/Lib/test/test_dict_version.py @@ -1,5 +1,5 @@ """ -Test implementation of the PEP 509: dictionary versionning. +Test implementation of the PEP 509: dictionary versioning. """ import unittest from test import support diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py index 1db73cc2d2220..3957077f5d612 100644 --- a/Lib/test/test_dtrace.py +++ b/Lib/test/test_dtrace.py @@ -34,7 +34,7 @@ def normalize_trace_output(output): return "\n".join(result) except (IndexError, ValueError): raise AssertionError( - "tracer produced unparseable output:\n{}".format(output) + "tracer produced unparsable output:\n{}".format(output) ) diff --git a/Lib/test/test_email/test_message.py b/Lib/test/test_email/test_message.py index 920a3d6a9cb91..4c754bf40fc30 100644 --- a/Lib/test/test_email/test_message.py +++ b/Lib/test/test_email/test_message.py @@ -433,7 +433,7 @@ class TestEmailMessageBase: --=== Content-Type: text/plain - Your message has bounced, ser. + Your message has bounced, sir. --=== Content-Type: message/rfc822 diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index cc59bf4c859d1..11c8f012896d3 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -247,7 +247,7 @@ def test_forced_io_encoding(self): def test_pre_initialization_api(self): """ - Checks some key parts of the C-API that need to work before the runtine + Checks some key parts of the C-API that need to work before the runtime is initialized (via Py_Initialize()). """ env = dict(os.environ, PYTHONPATH=os.pathsep.join(sys.path)) @@ -1106,7 +1106,7 @@ def test_init_setpath_config(self): 'base_prefix': '', 'exec_prefix': '', 'base_exec_prefix': '', - # overriden by PyConfig + # overridden by PyConfig 'program_name': 'conf_program_name', 'base_executable': 'conf_executable', 'executable': 'conf_executable', diff --git a/Lib/test/test_future.py b/Lib/test/test_future.py index 0f40357b3a731..afef14c460b69 100644 --- a/Lib/test/test_future.py +++ b/Lib/test/test_future.py @@ -322,7 +322,7 @@ def test_annotations(self): def test_fstring_debug_annotations(self): # f-strings with '=' don't round trip very well, so set the expected - # result explicitely. + # result explicitly. self.assertAnnotationEqual("f'{x=!r}'", expected="f'x={x!r}'") self.assertAnnotationEqual("f'{x=:}'", expected="f'x={x:}'") self.assertAnnotationEqual("f'{x=:.2f}'", expected="f'x={x:.2f}'") diff --git a/Lib/test/test_lltrace.py b/Lib/test/test_lltrace.py index 49fae81eefab9..e366966117952 100644 --- a/Lib/test/test_lltrace.py +++ b/Lib/test/test_lltrace.py @@ -12,7 +12,7 @@ def test_lltrace_does_not_crash_on_subscript_operator(self): # If this test fails, it will reproduce a crash reported as # bpo-34113. The crash happened at the command line console of # debug Python builds with __ltrace__ enabled (only possible in console), - # when the interal Python stack was negatively adjusted + # when the internal Python stack was negatively adjusted with open(support.TESTFN, 'w') as fd: self.addCleanup(os.unlink, support.TESTFN) fd.write(textwrap.dedent("""\ diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 375a1e4028631..a63cc0d9cc86e 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -494,7 +494,7 @@ def test_japanese(self): class TestMiscellaneous(unittest.TestCase): def test_defaults_UTF8(self): # Issue #18378: on (at least) macOS setting LC_CTYPE to "UTF-8" is - # valid. Futhermore LC_CTYPE=UTF is used by the UTF-8 locale coercing + # valid. Furthermore LC_CTYPE=UTF is used by the UTF-8 locale coercing # during interpreter startup (on macOS). import _locale import os diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index 0314938e30c14..183c157afdcd0 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -323,7 +323,7 @@ def test_match_common(self): self.assertFalse(P('b/py').match('b.py')) self.assertFalse(P('/a.py').match('b.py')) self.assertFalse(P('b.py/c').match('b.py')) - # Wilcard relative pattern. + # Wildcard relative pattern. self.assertTrue(P('b.py').match('*.py')) self.assertTrue(P('a/b.py').match('*.py')) self.assertTrue(P('/a/b.py').match('*.py')) @@ -1250,7 +1250,7 @@ def test_is_reserved(self): self.assertIs(False, P('/foo/bar').is_reserved()) # UNC paths are never reserved. self.assertIs(False, P('//my/share/nul/con/aux').is_reserved()) - # Case-insenstive DOS-device names are reserved. + # Case-insensitive DOS-device names are reserved. self.assertIs(True, P('nul').is_reserved()) self.assertIs(True, P('aux').is_reserved()) self.assertIs(True, P('prn').is_reserved()) diff --git a/Lib/test/test_strftime.py b/Lib/test/test_strftime.py index ec305e54ff24f..be43c49e40aa5 100644 --- a/Lib/test/test_strftime.py +++ b/Lib/test/test_strftime.py @@ -114,7 +114,7 @@ def strftest1(self, now): ) for e in expectations: - # musn't raise a value error + # mustn't raise a value error try: result = time.strftime(e[0], now) except ValueError as error: diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index f277c92c408d5..450e3920efa7c 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -409,7 +409,7 @@ def g456(): self.assertTrue(frame is sys._getframe()) # Verify that the captured thread frame is blocked in g456, called - # from f123. This is a litte tricky, since various bits of + # from f123. This is a little tricky, since various bits of # threading.py are also in the thread's call stack. frame = d.pop(thread_id) stack = traceback.extract_stack(frame) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index 6674edc21e9df..90a123828b709 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -435,8 +435,8 @@ def test_mktime(self): @unittest.skipUnless(platform.libc_ver()[0] != 'glibc', "disabled because of a bug in glibc. Issue #13309") def test_mktime_error(self): - # It may not be possible to reliably make mktime return error - # on all platfom. This will make sure that no other exception + # It may not be possible to reliably make mktime return an error + # on all platforms. This will make sure that no other exception # than OverflowError is raised for an extreme value. tt = time.gmtime(self.t) tzname = time.strftime('%Z', tt) diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py index ce03272ad30b2..d7587836aa8a3 100644 --- a/Lib/test/test_unparse.py +++ b/Lib/test/test_unparse.py @@ -408,7 +408,7 @@ def test_type_ignore(self): class CosmeticTestCase(ASTTestCase): - """Test if there are cosmetic issues caused by unnecesary additions""" + """Test if there are cosmetic issues caused by unnecessary additions""" def test_simple_expressions_parens(self): self.check_src_roundtrip("(a := b)") diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py index 5a3e59c3e9ef1..096833d1ebb35 100644 --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -1472,7 +1472,7 @@ def check_weak_del_and_len_while_iterating(self, dict, testcontext): o = Object(123456) with testcontext(): n = len(dict) - # Since underlaying dict is ordered, first item is popped + # Since underlying dict is ordered, first item is popped dict.pop(next(dict.keys())) self.assertEqual(len(dict), n - 1) dict[o] = o diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 4bf5d39e619f6..3e76e01c653f3 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -579,7 +579,7 @@ def testEnviron(self): # Test handler.environ as a dict expected = {} setup_testing_defaults(expected) - # Handler inherits os_environ variables which are not overriden + # Handler inherits os_environ variables which are not overridden # by SimpleHandler.add_cgi_vars() (SimpleHandler.base_env) for key, value in os_environ.items(): if key not in expected: diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 15c462ce80836..23c4cd5eff0a9 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -3272,7 +3272,7 @@ class MyElement(ET.Element): self._check_element_factory_class(MyElement) def test_element_factory_pure_python_subclass(self): - # Mimick SimpleTAL's behaviour (issue #16089): both versions of + # Mimic SimpleTAL's behaviour (issue #16089): both versions of # TreeBuilder should be able to cope with a subclass of the # pure Python Element class. base = ET._Element_Py diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index 83b618b749162..f714b773eeecc 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -559,7 +559,7 @@ def test_comparison(self): class BinaryTestCase(unittest.TestCase): - # XXX What should str(Binary(b"\xff")) return? I'm chosing "\xff" + # XXX What should str(Binary(b"\xff")) return? I'm choosing "\xff" # for now (i.e. interpreting the binary data as Latin-1-encoded # text). But this feels very unsatisfactory. Perhaps we should # only define repr(), and return r"Binary(b'\xff')" instead? diff --git a/Lib/threading.py b/Lib/threading.py index 0f2b9239ce399..a3cb245ab9606 100644 --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1466,7 +1466,7 @@ def _shutdown(): break for lock in locks: - # mimick Thread.join() + # mimic Thread.join() lock.acquire() lock.release() diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index f15f6c5e57a50..9049ca308f95a 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -2707,7 +2707,7 @@ def addtag_closest(self, newtag, x, y, halo=None, start=None): """Add tag NEWTAG to item which is closest to pixel at X, Y. If several match take the top-most. All items closer than HALO are considered overlapping (all are - closests). If START is specified the next below this tag is taken.""" + closest). If START is specified the next below this tag is taken.""" self.addtag(newtag, 'closest', x, y, halo, start) def addtag_enclosed(self, newtag, x1, y1, x2, y2): @@ -3302,7 +3302,7 @@ def add_command(self, cnf={}, **kw): self.add('command', cnf or kw) def add_radiobutton(self, cnf={}, **kw): - """Addd radio menu item.""" + """Add radio menu item.""" self.add('radiobutton', cnf or kw) def add_separator(self, cnf={}, **kw): @@ -3327,7 +3327,7 @@ def insert_command(self, index, cnf={}, **kw): self.insert(index, 'command', cnf or kw) def insert_radiobutton(self, index, cnf={}, **kw): - """Addd radio menu item at INDEX.""" + """Add radio menu item at INDEX.""" self.insert(index, 'radiobutton', cnf or kw) def insert_separator(self, index, cnf={}, **kw): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index ee5af82fd1b44..082da5d0c1a00 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -968,7 +968,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets readded at its previous position + # verify that the tab gets read at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index 3938efa3d423f..22cb61b7211c6 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -52,7 +52,7 @@ def addAsyncCleanup(self, func, /, *args, **kwargs): # We intentionally don't add inspect.iscoroutinefunction() check # for func argument because there is no way # to check for async function reliably: - # 1. It can be "async def func()" iself + # 1. It can be "async def func()" itself # 2. Class can implement "async def __call__()" method # 3. Regular "def func()" that returns awaitable object self.addCleanup(*(func, *args), **kwargs) diff --git a/Lib/unittest/test/testmock/testsealable.py b/Lib/unittest/test/testmock/testsealable.py index 11784c3678918..daba2b49b46f6 100644 --- a/Lib/unittest/test/testmock/testsealable.py +++ b/Lib/unittest/test/testmock/testsealable.py @@ -128,7 +128,7 @@ def test_integration_with_spec_att_definition(self): m.attr_sample2 def test_integration_with_spec_method_definition(self): - """You need to defin the methods, even if they are in the spec""" + """You need to define the methods, even if they are in the spec""" m = mock.Mock(SampleObject) m.method_sample1.return_value = 1 diff --git a/Lib/venv/scripts/common/Activate.ps1 b/Lib/venv/scripts/common/Activate.ps1 index a3bc6fb1f05bf..7bc3e7c698fc2 100644 --- a/Lib/venv/scripts/common/Activate.ps1 +++ b/Lib/venv/scripts/common/Activate.ps1 @@ -197,7 +197,7 @@ else { $Prompt = $pyvenvCfg['prompt']; } else { - Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virutal environment)" + Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)" Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'" $Prompt = Split-Path -Path $venvDir -Leaf } diff --git a/Lib/wsgiref/validate.py b/Lib/wsgiref/validate.py index 48ac0070549b3..6e16578dbb648 100644 --- a/Lib/wsgiref/validate.py +++ b/Lib/wsgiref/validate.py @@ -137,7 +137,7 @@ def validator(application): """ When applied between a WSGI server and a WSGI application, this - middleware will check for WSGI compliancy on a number of levels. + middleware will check for WSGI compliance on a number of levels. This middleware does not modify the request or response in any way, but will raise an AssertionError if anything seems off (except for a failure to close the application iterator, which diff --git a/Lib/zoneinfo/_zoneinfo.py b/Lib/zoneinfo/_zoneinfo.py index 9810637d3ef65..de68380792f17 100644 --- a/Lib/zoneinfo/_zoneinfo.py +++ b/Lib/zoneinfo/_zoneinfo.py @@ -338,7 +338,7 @@ def _utcoff_to_dstoff(trans_idx, utcoffsets, isdsts): comp_idx = trans_idx[i + 1] # If the following transition is also DST and we couldn't - # find the DST offset by this point, we're going ot have to + # find the DST offset by this point, we're going to have to # skip it and hope this transition gets assigned later if isdsts[comp_idx]: continue From webhook-mailer at python.org Thu Oct 7 11:49:56 2021 From: webhook-mailer at python.org (terryjreedy) Date: Thu, 07 Oct 2021 15:49:56 -0000 Subject: [Python-checkins] [3.10] Fix typos in the Lib directory (GH-28775) (GH-28804) Message-ID: https://github.com/python/cpython/commit/cfca4a67745cb9ed040fade3fd54f5002a6e75a6 commit: cfca4a67745cb9ed040fade3fd54f5002a6e75a6 branch: 3.10 author: Christian Clauss committer: terryjreedy date: 2021-10-07T11:49:47-04:00 summary: [3.10] Fix typos in the Lib directory (GH-28775) (GH-28804) Fix typos in the Lib directory as identified by codespell. Co-authored-by: Terry Jan Reedy . (cherry picked from commit 745c9d9dfc1ad6fdfdf1d07420c6273ff67fa5be) Co-authored-by: Christian Clauss files: M Lib/asyncio/events.py M Lib/asyncio/unix_events.py M Lib/ctypes/_aix.py M Lib/ctypes/test/test_structures.py M Lib/difflib.py M Lib/distutils/ccompiler.py M Lib/distutils/command/install.py M Lib/email/errors.py M Lib/html/parser.py M Lib/idlelib/ChangeLog M Lib/idlelib/idle_test/htest.py M Lib/idlelib/idle_test/mock_tk.py M Lib/idlelib/idle_test/test_multicall.py M Lib/idlelib/idle_test/test_pyparse.py M Lib/lib2to3/fixes/fix_metaclass.py M Lib/lib2to3/fixes/fix_paren.py M Lib/linecache.py M Lib/pickle.py M Lib/platform.py M Lib/sqlite3/test/dbapi.py M Lib/sysconfig.py M Lib/test/datetimetester.py M Lib/test/decimaltestdata/abs.decTest M Lib/test/decimaltestdata/extra.decTest M Lib/test/libregrtest/refleak.py M Lib/test/pickletester.py M Lib/test/support/threading_helper.py M Lib/test/test__xxsubinterpreters.py M Lib/test/test_asyncio/test_streams.py M Lib/test/test_asyncio/test_subprocess.py M Lib/test/test_asyncio/test_tasks.py M Lib/test/test_capi.py M Lib/test/test_collections.py M Lib/test/test_dataclasses.py M Lib/test/test_descr.py M Lib/test/test_dict.py M Lib/test/test_dict_version.py M Lib/test/test_dtrace.py M Lib/test/test_email/test_message.py M Lib/test/test_embed.py M Lib/test/test_exceptions.py M Lib/test/test_future.py M Lib/test/test_lltrace.py M Lib/test/test_locale.py M Lib/test/test_pathlib.py M Lib/test/test_strftime.py M Lib/test/test_sys.py M Lib/test/test_tempfile.py M Lib/test/test_time.py M Lib/test/test_unparse.py M Lib/test/test_weakref.py M Lib/test/test_wsgiref.py M Lib/test/test_xml_etree.py M Lib/test/test_xmlrpc.py M Lib/threading.py M Lib/tkinter/__init__.py M Lib/tkinter/test/test_ttk/test_widgets.py M Lib/unittest/async_case.py M Lib/unittest/test/testmock/testsealable.py M Lib/venv/scripts/common/Activate.ps1 M Lib/wsgiref/validate.py M Lib/zoneinfo/_zoneinfo.py diff --git a/Lib/asyncio/events.py b/Lib/asyncio/events.py index b966ad26bf467..7abaaca2d2b28 100644 --- a/Lib/asyncio/events.py +++ b/Lib/asyncio/events.py @@ -479,7 +479,7 @@ async def connect_read_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def connect_write_pipe(self, protocol_factory, pipe): @@ -492,7 +492,7 @@ async def connect_write_pipe(self, protocol_factory, pipe): # The reason to accept file-like object instead of just file descriptor # is: we need to own pipe and close it at transport finishing # Can got complicated errors if pass f.fileno(), - # close fd in pipe transport then close f and vise versa. + # close fd in pipe transport then close f and vice versa. raise NotImplementedError async def subprocess_shell(self, protocol_factory, cmd, *, diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index e4f445e95026b..4cef914b9fb9e 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1379,7 +1379,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classe requires it + # abstract base classes requires it return True def attach_loop(self, loop): diff --git a/Lib/ctypes/_aix.py b/Lib/ctypes/_aix.py index 26959d90a4dd6..fc3e95cbcc88a 100644 --- a/Lib/ctypes/_aix.py +++ b/Lib/ctypes/_aix.py @@ -163,7 +163,7 @@ def get_legacy(members): return member else: # 32-bit legacy names - both shr.o and shr4.o exist. - # shr.o is the preffered name so we look for shr.o first + # shr.o is the preferred name so we look for shr.o first # i.e., shr4.o is returned only when shr.o does not exist for name in ['shr.o', 'shr4.o']: member = get_one_match(re.escape(name), members) diff --git a/Lib/ctypes/test/test_structures.py b/Lib/ctypes/test/test_structures.py index 245cd94c5cdd9..97ad2b8ed8a50 100644 --- a/Lib/ctypes/test/test_structures.py +++ b/Lib/ctypes/test/test_structures.py @@ -443,7 +443,7 @@ def __del__(self): s = Test(1, 2, 3) # Test the StructUnionType_paramfunc() code path which copies the - # structure: if the stucture is larger than sizeof(void*). + # structure: if the structure is larger than sizeof(void*). self.assertGreater(sizeof(s), sizeof(c_void_p)) dll = CDLL(_ctypes_test.__file__) @@ -451,7 +451,7 @@ def __del__(self): func.argtypes = (Test,) func.restype = None func(s) - # bpo-37140: Passing the structure by refrence must not call + # bpo-37140: Passing the structure by reference must not call # its finalizer! self.assertEqual(finalizer_calls, []) self.assertEqual(s.first, 1) diff --git a/Lib/difflib.py b/Lib/difflib.py index 480bad2224c8e..afd8a0c7c5b61 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -62,7 +62,7 @@ class SequenceMatcher: notion, pairing up elements that appear uniquely in each sequence. That, and the method here, appear to yield more intuitive difference reports than does diff. This method appears to be the least vulnerable - to synching up on blocks of "junk lines", though (like blank lines in + to syncing up on blocks of "junk lines", though (like blank lines in ordinary text files, or maybe "

" lines in HTML files). That may be because this is the only method of the 3 that has a *concept* of "junk" . diff --git a/Lib/distutils/ccompiler.py b/Lib/distutils/ccompiler.py index b5ef143e72c56..4c47f2ed245d4 100644 --- a/Lib/distutils/ccompiler.py +++ b/Lib/distutils/ccompiler.py @@ -392,7 +392,7 @@ def _fix_compile_args(self, output_dir, macros, include_dirs): return output_dir, macros, include_dirs def _prep_compile(self, sources, output_dir, depends=None): - """Decide which souce files must be recompiled. + """Decide which source files must be recompiled. Determine the list of object files corresponding to 'sources', and figure out which ones really need to be recompiled. diff --git a/Lib/distutils/command/install.py b/Lib/distutils/command/install.py index 26696cfb9dcf9..01d5331a63069 100644 --- a/Lib/distutils/command/install.py +++ b/Lib/distutils/command/install.py @@ -31,7 +31,7 @@ # while making the sysconfig module the single point of truth. # This makes it easier for OS distributions where they need to # alter locations for packages installations in a single place. -# Note that this module is depracated (PEP 632); all consumers +# Note that this module is deprecated (PEP 632); all consumers # of this information should switch to using sysconfig directly. INSTALL_SCHEMES = {"unix_prefix": {}, "unix_home": {}, "nt": {}} @@ -43,7 +43,7 @@ sys_key = key sys_scheme = sysconfig._INSTALL_SCHEMES[sys_scheme_name] if key == "headers" and key not in sys_scheme: - # On POSIX-y platofrms, Python will: + # On POSIX-y platforms, Python will: # - Build from .h files in 'headers' (only there when # building CPython) # - Install .h files to 'include' diff --git a/Lib/email/errors.py b/Lib/email/errors.py index 1d258c34fc9d4..3ad0056554996 100644 --- a/Lib/email/errors.py +++ b/Lib/email/errors.py @@ -110,4 +110,4 @@ class NonASCIILocalPartDefect(HeaderDefect): # parsing messages decoded from binary. class InvalidDateDefect(HeaderDefect): - """Header has unparseable or invalid date""" + """Header has unparsable or invalid date""" diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 9e49effca1fcc..58f6bb3b1e932 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -405,7 +405,7 @@ def parse_endtag(self, i): tagname = namematch.group(1).lower() # consume and ignore other stuff between the name and the > # Note: this is not 100% correct, since we might have things like - # , but looking for > after tha name should cover + # , but looking for > after the name should cover # most of the cases and is much simpler gtpos = rawdata.find('>', namematch.end()) self.handle_endtag(tagname) diff --git a/Lib/idlelib/ChangeLog b/Lib/idlelib/ChangeLog index d7d7e1efdb1d3..c8960cfa535d0 100644 --- a/Lib/idlelib/ChangeLog +++ b/Lib/idlelib/ChangeLog @@ -1175,7 +1175,7 @@ Wed Mar 10 05:18:02 1999 Guido van Rossum classes in selected module methods of selected class - Sinlge clicking in a directory, module or class item updates the next + Single clicking in a directory, module or class item updates the next column with info about the selected item. Double clicking in a module, class or method item opens the file (and selects the clicked item if it is a class or method). diff --git a/Lib/idlelib/idle_test/htest.py b/Lib/idlelib/idle_test/htest.py index 1373b7642a6ea..666ff4cb84851 100644 --- a/Lib/idlelib/idle_test/htest.py +++ b/Lib/idlelib/idle_test/htest.py @@ -246,7 +246,7 @@ def _wrapper(parent): # htest # _object_browser_spec = { 'file': 'debugobj', 'kwds': {}, - 'msg': "Double click on items upto the lowest level.\n" + 'msg': "Double click on items up to the lowest level.\n" "Attributes of the objects and related information " "will be displayed side-by-side at each level." } @@ -255,7 +255,7 @@ def _wrapper(parent): # htest # 'file': 'pathbrowser', 'kwds': {}, 'msg': "Test for correct display of all paths in sys.path.\n" - "Toggle nested items upto the lowest level.\n" + "Toggle nested items up to the lowest level.\n" "Double clicking on an item prints a traceback\n" "for an exception that is ignored." } @@ -341,7 +341,7 @@ def _wrapper(parent): # htest # 'file': 'tree', 'kwds': {}, 'msg': "The canvas is scrollable.\n" - "Click on folders upto to the lowest level." + "Click on folders up to to the lowest level." } _undo_delegator_spec = { diff --git a/Lib/idlelib/idle_test/mock_tk.py b/Lib/idlelib/idle_test/mock_tk.py index db583553838fb..8304734b847a8 100644 --- a/Lib/idlelib/idle_test/mock_tk.py +++ b/Lib/idlelib/idle_test/mock_tk.py @@ -79,7 +79,7 @@ def tearDownClass(cls): --- For 'ask' functions, set func.result return value before calling the method that uses the message function. When messagebox functions are the - only gui alls in a method, this replacement makes the method gui-free, + only GUI calls in a method, this replacement makes the method GUI-free, """ askokcancel = Mbox_func() # True or False askquestion = Mbox_func() # 'yes' or 'no' diff --git a/Lib/idlelib/idle_test/test_multicall.py b/Lib/idlelib/idle_test/test_multicall.py index ba582bb3ca51b..b3a3bfb88f9c3 100644 --- a/Lib/idlelib/idle_test/test_multicall.py +++ b/Lib/idlelib/idle_test/test_multicall.py @@ -37,7 +37,7 @@ def test_init(self): def test_yview(self): # Added for tree.wheel_event - # (it depends on yview to not be overriden) + # (it depends on yview to not be overridden) mc = self.mc self.assertIs(mc.yview, Text.yview) mctext = self.mc(self.root) diff --git a/Lib/idlelib/idle_test/test_pyparse.py b/Lib/idlelib/idle_test/test_pyparse.py index fb5726db1d821..384db566ac76c 100644 --- a/Lib/idlelib/idle_test/test_pyparse.py +++ b/Lib/idlelib/idle_test/test_pyparse.py @@ -284,7 +284,7 @@ def test_get_num_lines_in_stmt(self): tests = ( TestInfo('[x for x in a]\n', 1), # Closed on one line. TestInfo('[x\nfor x in a\n', 2), # Not closed. - TestInfo('[x\\\nfor x in a\\\n', 2), # "", uneeded backslashes. + TestInfo('[x\\\nfor x in a\\\n', 2), # "", unneeded backslashes. TestInfo('[x\nfor x in a\n]\n', 3), # Closed on multi-line. TestInfo('\n"""Docstring comment L1"""\nL2\nL3\nL4\n', 1), TestInfo('\n"""Docstring comment L1\nL2"""\nL3\nL4\n', 1), diff --git a/Lib/lib2to3/fixes/fix_metaclass.py b/Lib/lib2to3/fixes/fix_metaclass.py index d1cd10d327587..fe547b2228072 100644 --- a/Lib/lib2to3/fixes/fix_metaclass.py +++ b/Lib/lib2to3/fixes/fix_metaclass.py @@ -51,7 +51,7 @@ def fixup_parse_tree(cls_node): # already in the preferred format, do nothing return - # !%@#! oneliners have no suite node, we have to fake one up + # !%@#! one-liners have no suite node, we have to fake one up for i, node in enumerate(cls_node.children): if node.type == token.COLON: break diff --git a/Lib/lib2to3/fixes/fix_paren.py b/Lib/lib2to3/fixes/fix_paren.py index b205aa7e1e93f..df3da5f5232c9 100644 --- a/Lib/lib2to3/fixes/fix_paren.py +++ b/Lib/lib2to3/fixes/fix_paren.py @@ -1,4 +1,4 @@ -"""Fixer that addes parentheses where they are required +"""Fixer that adds parentheses where they are required This converts ``[x for x in 1, 2]`` to ``[x for x in (1, 2)]``.""" diff --git a/Lib/linecache.py b/Lib/linecache.py index 513b17e999880..23191d6501d2a 100644 --- a/Lib/linecache.py +++ b/Lib/linecache.py @@ -154,7 +154,7 @@ def lazycache(filename, module_globals): :return: True if a lazy load is registered in the cache, otherwise False. To register such a load a module loader with a - get_source method must be found, the filename must be a cachable + get_source method must be found, the filename must be a cacheable filename, and the filename must not be already cached. """ if filename in cache: diff --git a/Lib/pickle.py b/Lib/pickle.py index 5ab312f2acaee..e7f30f226101f 100644 --- a/Lib/pickle.py +++ b/Lib/pickle.py @@ -1173,7 +1173,7 @@ def __init__(self, file, *, fix_imports=True, used in Python 3. The *encoding* and *errors* tell pickle how to decode 8-bit string instances pickled by Python 2; these default to 'ASCII' and 'strict', respectively. *encoding* can be - 'bytes' to read theses 8-bit string instances as bytes objects. + 'bytes' to read these 8-bit string instances as bytes objects. """ self._buffers = iter(buffers) if buffers is not None else None self._file_readline = file.readline diff --git a/Lib/platform.py b/Lib/platform.py index 134fbae6b1cc7..e32f9c11cdbf1 100755 --- a/Lib/platform.py +++ b/Lib/platform.py @@ -1262,7 +1262,7 @@ def platform(aliased=0, terse=0): def _parse_os_release(lines): # These fields are mandatory fields with well-known defaults - # in pratice all Linux distributions override NAME, ID, and PRETTY_NAME. + # in practice all Linux distributions override NAME, ID, and PRETTY_NAME. info = { "NAME": "Linux", "ID": "linux", diff --git a/Lib/sqlite3/test/dbapi.py b/Lib/sqlite3/test/dbapi.py index a8dcd7e8bfedd..e332184a7d107 100644 --- a/Lib/sqlite3/test/dbapi.py +++ b/Lib/sqlite3/test/dbapi.py @@ -454,7 +454,7 @@ def test_fetchone_no_statement(self): self.assertEqual(row, None) def test_array_size(self): - # must default ot 1 + # must default to 1 self.assertEqual(self.cu.arraysize, 1) # now set to 2 diff --git a/Lib/sysconfig.py b/Lib/sysconfig.py index 95b48f6429d5f..daf9f000060a3 100644 --- a/Lib/sysconfig.py +++ b/Lib/sysconfig.py @@ -184,7 +184,7 @@ def is_python_build(check_home=False): if _PYTHON_BUILD: for scheme in ('posix_prefix', 'posix_home'): - # On POSIX-y platofrms, Python will: + # On POSIX-y platforms, Python will: # - Build from .h files in 'headers' (which is only added to the # scheme when building CPython) # - Install .h files to 'include' diff --git a/Lib/test/datetimetester.py b/Lib/test/datetimetester.py index 6414f1ace3fed..9f551d9b9748d 100644 --- a/Lib/test/datetimetester.py +++ b/Lib/test/datetimetester.py @@ -4064,7 +4064,7 @@ def test_even_more_compare(self): self.assertEqual(t1, t1) self.assertEqual(t2, t2) - # Equal afer adjustment. + # Equal after adjustment. t1 = self.theclass(1, 12, 31, 23, 59, tzinfo=FixedOffset(1, "")) t2 = self.theclass(2, 1, 1, 3, 13, tzinfo=FixedOffset(3*60+13+2, "")) self.assertEqual(t1, t2) @@ -4903,7 +4903,7 @@ def test_easy(self): # OTOH, these fail! Don't enable them. The difficulty is that # the edge case tests assume that every hour is representable in # the "utc" class. This is always true for a fixed-offset tzinfo - # class (lke utc_real and utc_fake), but not for Eastern or Central. + # class (like utc_real and utc_fake), but not for Eastern or Central. # For these adjacent DST-aware time zones, the range of time offsets # tested ends up creating hours in the one that aren't representable # in the other. For the same reason, we would see failures in the diff --git a/Lib/test/decimaltestdata/abs.decTest b/Lib/test/decimaltestdata/abs.decTest index 01f73d7766648..569b8fcd84ab6 100644 --- a/Lib/test/decimaltestdata/abs.decTest +++ b/Lib/test/decimaltestdata/abs.decTest @@ -20,7 +20,7 @@ version: 2.59 -- This set of tests primarily tests the existence of the operator. --- Additon, subtraction, rounding, and more overflows are tested +-- Addition, subtraction, rounding, and more overflows are tested -- elsewhere. precision: 9 diff --git a/Lib/test/decimaltestdata/extra.decTest b/Lib/test/decimaltestdata/extra.decTest index b630d8e3f9d45..31291202a35e7 100644 --- a/Lib/test/decimaltestdata/extra.decTest +++ b/Lib/test/decimaltestdata/extra.decTest @@ -156,7 +156,7 @@ extr1302 fma -Inf 0E-456 sNaN148 -> NaN Invalid_operation -- max/min/max_mag/min_mag bug in 2.5.2/2.6/3.0: max(NaN, finite) gave -- incorrect answers when the finite number required rounding; similarly --- for the other thre functions +-- for the other three functions maxexponent: 999 minexponent: -999 precision: 6 diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index 7c7086a806b1c..b56a84856d14f 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -112,7 +112,7 @@ def get_pooled_int(value): # These checkers return False on success, True on failure def check_rc_deltas(deltas): - # Checker for reference counters and memomry blocks. + # Checker for reference counters and memory blocks. # # bpo-30776: Try to ignore false positives: # diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 25283f8c538d8..3e2c781caa11a 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -3723,7 +3723,7 @@ class MyClass: self.assertEqual(new_f, 5) self.assertEqual(some_str, 'some str') - # math.log does not have its usual reducer overriden, so the + # math.log does not have its usual reducer overridden, so the # custom reduction callback should silently direct the pickler # to the default pickling by attribute, by returning # NotImplemented @@ -3740,7 +3740,7 @@ class MyClass: def test_reducer_override_no_reference_cycle(self): # bpo-39492: reducer_override used to induce a spurious reference cycle # inside the Pickler object, that could prevent all serialized objects - # from being garbage-collected without explicity invoking gc.collect. + # from being garbage-collected without explicitly invoking gc.collect. for proto in range(0, pickle.HIGHEST_PROTOCOL + 1): with self.subTest(proto=proto): diff --git a/Lib/test/support/threading_helper.py b/Lib/test/support/threading_helper.py index 0632577cdb303..92a64e8354acb 100644 --- a/Lib/test/support/threading_helper.py +++ b/Lib/test/support/threading_helper.py @@ -157,7 +157,7 @@ class catch_threading_exception: Context manager catching threading.Thread exception using threading.excepthook. - Attributes set when an exception is catched: + Attributes set when an exception is caught: * exc_type * exc_value diff --git a/Lib/test/test__xxsubinterpreters.py b/Lib/test/test__xxsubinterpreters.py index 81bce2e620421..177a8a64a4329 100644 --- a/Lib/test/test__xxsubinterpreters.py +++ b/Lib/test/test__xxsubinterpreters.py @@ -1221,7 +1221,7 @@ def test_channel_list_interpreters_basic(self): import _xxsubinterpreters as _interpreters obj = _interpreters.channel_recv({cid}) """)) - # Test for channel that has boths ends associated to an interpreter. + # Test for channel that has both ends associated to an interpreter. send_interps = interpreters.channel_list_interpreters(cid, send=True) recv_interps = interpreters.channel_list_interpreters(cid, send=False) self.assertEqual(send_interps, [interp0]) diff --git a/Lib/test/test_asyncio/test_streams.py b/Lib/test/test_asyncio/test_streams.py index 6eaa289944218..a6ea24ceceda2 100644 --- a/Lib/test/test_asyncio/test_streams.py +++ b/Lib/test/test_asyncio/test_streams.py @@ -711,7 +711,7 @@ def test_read_all_from_pipe_reader(self): # See asyncio issue 168. This test is derived from the example # subprocess_attach_read_pipe.py, but we configure the # StreamReader's limit so that twice it is less than the size - # of the data writter. Also we must explicitly attach a child + # of the data writer. Also we must explicitly attach a child # watcher to the event loop. code = """\ diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 3cf88188ecf3b..14fa6dd76f9ca 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -228,7 +228,7 @@ def prepare_broken_pipe_test(self): # buffer large enough to feed the whole pipe buffer large_data = b'x' * support.PIPE_MAX_SIZE - # the program ends before the stdin can be feeded + # the program ends before the stdin can be fed proc = self.loop.run_until_complete( asyncio.create_subprocess_exec( sys.executable, '-c', 'pass', diff --git a/Lib/test/test_asyncio/test_tasks.py b/Lib/test/test_asyncio/test_tasks.py index 9498c7251f6f6..ff627a6980ea0 100644 --- a/Lib/test/test_asyncio/test_tasks.py +++ b/Lib/test/test_asyncio/test_tasks.py @@ -3673,7 +3673,7 @@ def test_run_coroutine_threadsafe_with_timeout(self): self.assertTrue(task.done()) def test_run_coroutine_threadsafe_task_cancelled(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task is cancelled.""" callback = lambda: self.target(cancel=True) future = self.loop.run_in_executor(None, callback) @@ -3681,7 +3681,7 @@ def test_run_coroutine_threadsafe_task_cancelled(self): self.loop.run_until_complete(future) def test_run_coroutine_threadsafe_task_factory_exception(self): - """Test coroutine submission from a tread to an event loop + """Test coroutine submission from a thread to an event loop when the task factory raise an exception.""" def task_factory(loop, coro): diff --git a/Lib/test/test_capi.py b/Lib/test/test_capi.py index db26b9b5ddedb..6ed5ecdeca5e2 100644 --- a/Lib/test/test_capi.py +++ b/Lib/test/test_capi.py @@ -264,7 +264,7 @@ def test_return_result_with_error(self): def test_getitem_with_error(self): # Test _Py_CheckSlotResult(). Raise an exception and then calls - # PyObject_GetItem(): check that the assertion catchs the bug. + # PyObject_GetItem(): check that the assertion catches the bug. # PyObject_GetItem() must not be called with an exception set. code = textwrap.dedent(""" import _testcapi diff --git a/Lib/test/test_collections.py b/Lib/test/test_collections.py index 1f659d7d604e3..75af29b2dc72d 100644 --- a/Lib/test/test_collections.py +++ b/Lib/test/test_collections.py @@ -1594,7 +1594,7 @@ def assertSameSet(self, s1, s2): self.assertSetEqual(set(s1), set(s2)) def test_Set_from_iterable(self): - """Verify _from_iterable overriden to an instance method works.""" + """Verify _from_iterable overridden to an instance method works.""" class SetUsingInstanceFromIterable(MutableSet): def __init__(self, values, created_by): if not created_by: diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index 33c9fcd165621..a1d9112135af3 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -3695,7 +3695,7 @@ class B: with self.assertRaisesRegex(TypeError, msg): B(3, 4, 5) - # Explicitely make a field that follows KW_ONLY be non-keyword-only. + # Explicitly make a field that follows KW_ONLY be non-keyword-only. @dataclass class C: a: int diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index b8862fd6fcace..3df69ba11407a 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -5723,7 +5723,7 @@ class A(metaclass=M): def test_incomplete_super(self): """ - Attrubute lookup on a super object must be aware that + Attribute lookup on a super object must be aware that its target type can be uninitialized (type->tp_mro == NULL). """ class M(DebugHelperMeta): diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index 666cd81e68d81..54d100288cb62 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -1051,7 +1051,7 @@ def test_splittable_pop(self): @support.cpython_only def test_splittable_pop_pending(self): - """pop a pending key in a splitted table should not crash""" + """pop a pending key in a split table should not crash""" a, b = self.make_shared_key_dict(2) a['a'] = 4 @@ -1398,7 +1398,7 @@ def test_reversed(self): self.assertRaises(StopIteration, next, r) def test_reverse_iterator_for_empty_dict(self): - # bpo-38525: revered iterator should work properly + # bpo-38525: reversed iterator should work properly # empty dict is directly used for reference count test self.assertEqual(list(reversed({})), []) diff --git a/Lib/test/test_dict_version.py b/Lib/test/test_dict_version.py index 8cdccad0d79ab..243084c75c42b 100644 --- a/Lib/test/test_dict_version.py +++ b/Lib/test/test_dict_version.py @@ -1,5 +1,5 @@ """ -Test implementation of the PEP 509: dictionary versionning. +Test implementation of the PEP 509: dictionary versioning. """ import unittest from test.support import import_helper diff --git a/Lib/test/test_dtrace.py b/Lib/test/test_dtrace.py index 1db73cc2d2220..3957077f5d612 100644 --- a/Lib/test/test_dtrace.py +++ b/Lib/test/test_dtrace.py @@ -34,7 +34,7 @@ def normalize_trace_output(output): return "\n".join(result) except (IndexError, ValueError): raise AssertionError( - "tracer produced unparseable output:\n{}".format(output) + "tracer produced unparsable output:\n{}".format(output) ) diff --git a/Lib/test/test_email/test_message.py b/Lib/test/test_email/test_message.py index 920a3d6a9cb91..4c754bf40fc30 100644 --- a/Lib/test/test_email/test_message.py +++ b/Lib/test/test_email/test_message.py @@ -433,7 +433,7 @@ class TestEmailMessageBase: --=== Content-Type: text/plain - Your message has bounced, ser. + Your message has bounced, sir. --=== Content-Type: message/rfc822 diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 0a66e7f6b9a82..da9b555522f4f 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -250,7 +250,7 @@ def test_forced_io_encoding(self): def test_pre_initialization_api(self): """ - Checks some key parts of the C-API that need to work before the runtine + Checks some key parts of the C-API that need to work before the runtime is initialized (via Py_Initialize()). """ env = dict(os.environ, PYTHONPATH=os.pathsep.join(sys.path)) @@ -1157,7 +1157,7 @@ def test_init_setpath_config(self): 'base_prefix': '', 'exec_prefix': '', 'base_exec_prefix': '', - # overriden by PyConfig + # overridden by PyConfig 'program_name': 'conf_program_name', 'base_executable': 'conf_executable', 'executable': 'conf_executable', diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 143b856280028..d04e5f5573ca9 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -2281,7 +2281,7 @@ def test_range_of_offsets(self): abcdefg SyntaxError: bad bad """)), - # End offset pass the source lenght + # End offset pass the source length (("bad.py", 1, 2, "abcdefg", 1, 100), dedent( """ diff --git a/Lib/test/test_future.py b/Lib/test/test_future.py index 4ef11232a3368..5a3944e69e640 100644 --- a/Lib/test/test_future.py +++ b/Lib/test/test_future.py @@ -329,7 +329,7 @@ def test_annotations(self): def test_fstring_debug_annotations(self): # f-strings with '=' don't round trip very well, so set the expected - # result explicitely. + # result explicitly. self.assertAnnotationEqual("f'{x=!r}'", expected="f'x={x!r}'") self.assertAnnotationEqual("f'{x=:}'", expected="f'x={x:}'") self.assertAnnotationEqual("f'{x=:.2f}'", expected="f'x={x:.2f}'") diff --git a/Lib/test/test_lltrace.py b/Lib/test/test_lltrace.py index 8f1a92e5c725c..06e33f4c4c2f3 100644 --- a/Lib/test/test_lltrace.py +++ b/Lib/test/test_lltrace.py @@ -12,7 +12,7 @@ def test_lltrace_does_not_crash_on_subscript_operator(self): # If this test fails, it will reproduce a crash reported as # bpo-34113. The crash happened at the command line console of # debug Python builds with __ltrace__ enabled (only possible in console), - # when the interal Python stack was negatively adjusted + # when the internal Python stack was negatively adjusted with open(os_helper.TESTFN, 'w', encoding='utf-8') as fd: self.addCleanup(os_helper.unlink, os_helper.TESTFN) fd.write(textwrap.dedent("""\ diff --git a/Lib/test/test_locale.py b/Lib/test/test_locale.py index 2c788f2dfa65e..f844e62ca2e72 100644 --- a/Lib/test/test_locale.py +++ b/Lib/test/test_locale.py @@ -496,7 +496,7 @@ def test_japanese(self): class TestMiscellaneous(unittest.TestCase): def test_defaults_UTF8(self): # Issue #18378: on (at least) macOS setting LC_CTYPE to "UTF-8" is - # valid. Futhermore LC_CTYPE=UTF is used by the UTF-8 locale coercing + # valid. Furthermore LC_CTYPE=UTF is used by the UTF-8 locale coercing # during interpreter startup (on macOS). import _locale import os diff --git a/Lib/test/test_pathlib.py b/Lib/test/test_pathlib.py index e716f4d385932..5f6d9f47d13d7 100644 --- a/Lib/test/test_pathlib.py +++ b/Lib/test/test_pathlib.py @@ -324,7 +324,7 @@ def test_match_common(self): self.assertFalse(P('b/py').match('b.py')) self.assertFalse(P('/a.py').match('b.py')) self.assertFalse(P('b.py/c').match('b.py')) - # Wilcard relative pattern. + # Wildcard relative pattern. self.assertTrue(P('b.py').match('*.py')) self.assertTrue(P('a/b.py').match('*.py')) self.assertTrue(P('/a/b.py').match('*.py')) @@ -1284,7 +1284,7 @@ def test_is_reserved(self): self.assertIs(False, P('/foo/bar').is_reserved()) # UNC paths are never reserved. self.assertIs(False, P('//my/share/nul/con/aux').is_reserved()) - # Case-insenstive DOS-device names are reserved. + # Case-insensitive DOS-device names are reserved. self.assertIs(True, P('nul').is_reserved()) self.assertIs(True, P('aux').is_reserved()) self.assertIs(True, P('prn').is_reserved()) diff --git a/Lib/test/test_strftime.py b/Lib/test/test_strftime.py index ec305e54ff24f..be43c49e40aa5 100644 --- a/Lib/test/test_strftime.py +++ b/Lib/test/test_strftime.py @@ -114,7 +114,7 @@ def strftest1(self, now): ) for e in expectations: - # musn't raise a value error + # mustn't raise a value error try: result = time.strftime(e[0], now) except ValueError as error: diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 8717def2a544a..35da72c432b0a 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -379,7 +379,7 @@ def g456(): self.assertTrue(frame is sys._getframe()) # Verify that the captured thread frame is blocked in g456, called - # from f123. This is a litte tricky, since various bits of + # from f123. This is a little tricky, since various bits of # threading.py are also in the thread's call stack. frame = d.pop(thread_id) stack = traceback.extract_stack(frame) @@ -446,7 +446,7 @@ def g456(): self.assertEqual((None, None, None), d.pop(main_id)) # Verify that the captured thread frame is blocked in g456, called - # from f123. This is a litte tricky, since various bits of + # from f123. This is a little tricky, since various bits of # threading.py are also in the thread's call stack. exc_type, exc_value, exc_tb = d.pop(thread_id) stack = traceback.extract_stack(exc_tb.tb_frame) diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index f1d483733e267..96946a281a490 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -1435,7 +1435,7 @@ def test_explict_cleanup_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") temp_dir.cleanup() self.assertFalse( temp_path.exists(), @@ -1494,7 +1494,7 @@ def test_del_on_collection_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") def test_del_on_shutdown(self): # A TemporaryDirectory may be cleaned up during shutdown @@ -1559,7 +1559,7 @@ def test_del_on_shutdown_ignore_errors(self): self.assertEqual( temp_path.exists(), sys.platform.startswith("win"), - f"TemporaryDirectory {temp_path!s} existance state unexpected") + f"TemporaryDirectory {temp_path!s} existence state unexpected") err = err.decode('utf-8', 'backslashreplace') self.assertNotIn("Exception", err) self.assertNotIn("Error", err) diff --git a/Lib/test/test_time.py b/Lib/test/test_time.py index db929bd881778..875615ad51007 100644 --- a/Lib/test/test_time.py +++ b/Lib/test/test_time.py @@ -436,8 +436,8 @@ def test_mktime(self): @unittest.skipUnless(platform.libc_ver()[0] != 'glibc', "disabled because of a bug in glibc. Issue #13309") def test_mktime_error(self): - # It may not be possible to reliably make mktime return error - # on all platfom. This will make sure that no other exception + # It may not be possible to reliably make mktime return an error + # on all platforms. This will make sure that no other exception # than OverflowError is raised for an extreme value. tt = time.gmtime(self.t) tzname = time.strftime('%Z', tt) diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py index 9f67b49f3a6b2..33e1149bfd360 100644 --- a/Lib/test/test_unparse.py +++ b/Lib/test/test_unparse.py @@ -408,7 +408,7 @@ def test_type_ignore(self): class CosmeticTestCase(ASTTestCase): - """Test if there are cosmetic issues caused by unnecesary additions""" + """Test if there are cosmetic issues caused by unnecessary additions""" def test_simple_expressions_parens(self): self.check_src_roundtrip("(a := b)") diff --git a/Lib/test/test_weakref.py b/Lib/test/test_weakref.py index 5a3e59c3e9ef1..096833d1ebb35 100644 --- a/Lib/test/test_weakref.py +++ b/Lib/test/test_weakref.py @@ -1472,7 +1472,7 @@ def check_weak_del_and_len_while_iterating(self, dict, testcontext): o = Object(123456) with testcontext(): n = len(dict) - # Since underlaying dict is ordered, first item is popped + # Since underlying dict is ordered, first item is popped dict.pop(next(dict.keys())) self.assertEqual(len(dict), n - 1) dict[o] = o diff --git a/Lib/test/test_wsgiref.py b/Lib/test/test_wsgiref.py index 93ca6b99a92c9..42094f467731d 100644 --- a/Lib/test/test_wsgiref.py +++ b/Lib/test/test_wsgiref.py @@ -580,7 +580,7 @@ def testEnviron(self): # Test handler.environ as a dict expected = {} setup_testing_defaults(expected) - # Handler inherits os_environ variables which are not overriden + # Handler inherits os_environ variables which are not overridden # by SimpleHandler.add_cgi_vars() (SimpleHandler.base_env) for key, value in os_environ.items(): if key not in expected: diff --git a/Lib/test/test_xml_etree.py b/Lib/test/test_xml_etree.py index 5a8824a78ffa4..285559a872a65 100644 --- a/Lib/test/test_xml_etree.py +++ b/Lib/test/test_xml_etree.py @@ -3331,7 +3331,7 @@ class MyElement(ET.Element): self._check_element_factory_class(MyElement) def test_element_factory_pure_python_subclass(self): - # Mimick SimpleTAL's behaviour (issue #16089): both versions of + # Mimic SimpleTAL's behaviour (issue #16089): both versions of # TreeBuilder should be able to cope with a subclass of the # pure Python Element class. base = ET._Element_Py diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index 85e27ad4631d2..1f06f5fdf483e 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -561,7 +561,7 @@ def test_comparison(self): class BinaryTestCase(unittest.TestCase): - # XXX What should str(Binary(b"\xff")) return? I'm chosing "\xff" + # XXX What should str(Binary(b"\xff")) return? I'm choosing "\xff" # for now (i.e. interpreting the binary data as Latin-1-encoded # text). But this feels very unsatisfactory. Perhaps we should # only define repr(), and return r"Binary(b'\xff')" instead? diff --git a/Lib/threading.py b/Lib/threading.py index 3e9577d8a5b14..2d8974291360d 100644 --- a/Lib/threading.py +++ b/Lib/threading.py @@ -1556,7 +1556,7 @@ def _shutdown(): break for lock in locks: - # mimick Thread.join() + # mimic Thread.join() lock.acquire() lock.release() diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index 552fc54a4934d..fa88448a2c071 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -2736,7 +2736,7 @@ def addtag_closest(self, newtag, x, y, halo=None, start=None): """Add tag NEWTAG to item which is closest to pixel at X, Y. If several match take the top-most. All items closer than HALO are considered overlapping (all are - closests). If START is specified the next below this tag is taken.""" + closest). If START is specified the next below this tag is taken.""" self.addtag(newtag, 'closest', x, y, halo, start) def addtag_enclosed(self, newtag, x1, y1, x2, y2): @@ -3331,7 +3331,7 @@ def add_command(self, cnf={}, **kw): self.add('command', cnf or kw) def add_radiobutton(self, cnf={}, **kw): - """Addd radio menu item.""" + """Add radio menu item.""" self.add('radiobutton', cnf or kw) def add_separator(self, cnf={}, **kw): @@ -3356,7 +3356,7 @@ def insert_command(self, index, cnf={}, **kw): self.insert(index, 'command', cnf or kw) def insert_radiobutton(self, index, cnf={}, **kw): - """Addd radio menu item at INDEX.""" + """Add radio menu item at INDEX.""" self.insert(index, 'radiobutton', cnf or kw) def insert_separator(self, index, cnf={}, **kw): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index ee5af82fd1b44..082da5d0c1a00 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -968,7 +968,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets readded at its previous position + # verify that the tab gets read at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) diff --git a/Lib/unittest/async_case.py b/Lib/unittest/async_case.py index c48380e5b1808..4f9a80be80c5a 100644 --- a/Lib/unittest/async_case.py +++ b/Lib/unittest/async_case.py @@ -52,7 +52,7 @@ def addAsyncCleanup(self, func, /, *args, **kwargs): # We intentionally don't add inspect.iscoroutinefunction() check # for func argument because there is no way # to check for async function reliably: - # 1. It can be "async def func()" iself + # 1. It can be "async def func()" itself # 2. Class can implement "async def __call__()" method # 3. Regular "def func()" that returns awaitable object self.addCleanup(*(func, *args), **kwargs) diff --git a/Lib/unittest/test/testmock/testsealable.py b/Lib/unittest/test/testmock/testsealable.py index 11784c3678918..daba2b49b46f6 100644 --- a/Lib/unittest/test/testmock/testsealable.py +++ b/Lib/unittest/test/testmock/testsealable.py @@ -128,7 +128,7 @@ def test_integration_with_spec_att_definition(self): m.attr_sample2 def test_integration_with_spec_method_definition(self): - """You need to defin the methods, even if they are in the spec""" + """You need to define the methods, even if they are in the spec""" m = mock.Mock(SampleObject) m.method_sample1.return_value = 1 diff --git a/Lib/venv/scripts/common/Activate.ps1 b/Lib/venv/scripts/common/Activate.ps1 index 51fc55c4ec7cb..eeea3583fa130 100644 --- a/Lib/venv/scripts/common/Activate.ps1 +++ b/Lib/venv/scripts/common/Activate.ps1 @@ -202,7 +202,7 @@ else { $Prompt = $pyvenvCfg['prompt']; } else { - Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virutal environment)" + Write-Verbose " Setting prompt based on parent's directory's name. (Is the directory name passed to venv module when creating the virtual environment)" Write-Verbose " Got leaf-name of $VenvDir='$(Split-Path -Path $venvDir -Leaf)'" $Prompt = Split-Path -Path $venvDir -Leaf } diff --git a/Lib/wsgiref/validate.py b/Lib/wsgiref/validate.py index 48ac0070549b3..6e16578dbb648 100644 --- a/Lib/wsgiref/validate.py +++ b/Lib/wsgiref/validate.py @@ -137,7 +137,7 @@ def validator(application): """ When applied between a WSGI server and a WSGI application, this - middleware will check for WSGI compliancy on a number of levels. + middleware will check for WSGI compliance on a number of levels. This middleware does not modify the request or response in any way, but will raise an AssertionError if anything seems off (except for a failure to close the application iterator, which diff --git a/Lib/zoneinfo/_zoneinfo.py b/Lib/zoneinfo/_zoneinfo.py index 9810637d3ef65..de68380792f17 100644 --- a/Lib/zoneinfo/_zoneinfo.py +++ b/Lib/zoneinfo/_zoneinfo.py @@ -338,7 +338,7 @@ def _utcoff_to_dstoff(trans_idx, utcoffsets, isdsts): comp_idx = trans_idx[i + 1] # If the following transition is also DST and we couldn't - # find the DST offset by this point, we're going ot have to + # find the DST offset by this point, we're going to have to # skip it and hope this transition gets assigned later if isdsts[comp_idx]: continue From webhook-mailer at python.org Thu Oct 7 14:00:41 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 18:00:41 -0000 Subject: [Python-checkins] Remove draft notice on the 3.10 What's new document (GH-28806) Message-ID: https://github.com/python/cpython/commit/32485cecab0111a858055b7a60df3b9903b162e3 commit: 32485cecab0111a858055b7a60df3b9903b162e3 branch: main author: Pablo Galindo Salgado committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T11:00:29-07:00 summary: Remove draft notice on the 3.10 What's new document (GH-28806) Automerge-Triggered-By: GH:pablogsal files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index b063255bc3e9e..266a80be3c96c 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -50,13 +50,6 @@ This article explains the new features in Python 3.10, compared to 3.9. For full details, see the :ref:`changelog `. -.. note:: - - Prerelease users should be aware that this document is currently in draft - form. It will be updated substantially as Python 3.10 moves towards release, - so it's worth checking back even after reading earlier versions. - - Summary -- Release highlights ============================= From webhook-mailer at python.org Thu Oct 7 14:25:36 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 18:25:36 -0000 Subject: [Python-checkins] [3.10] Remove draft notice on the 3.10 What's new document (GH-28806) (GH-28807) Message-ID: https://github.com/python/cpython/commit/86bf45e69e22bb5950f1d47c670dfb36bbd45dd0 commit: 86bf45e69e22bb5950f1d47c670dfb36bbd45dd0 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T11:25:26-07:00 summary: [3.10] Remove draft notice on the 3.10 What's new document (GH-28806) (GH-28807) Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 32485cecab0111a858055b7a60df3b9903b162e3) Co-authored-by: Pablo Galindo Salgado files: M Doc/whatsnew/3.10.rst diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 7e8ae42085001..ddcefb408b7f8 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -50,13 +50,6 @@ This article explains the new features in Python 3.10, compared to 3.9. For full details, see the :ref:`changelog `. -.. note:: - - Prerelease users should be aware that this document is currently in draft - form. It will be updated substantially as Python 3.10 moves towards release, - so it's worth checking back even after reading earlier versions. - - Summary -- Release highlights ============================= From webhook-mailer at python.org Thu Oct 7 15:19:20 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 07 Oct 2021 19:19:20 -0000 Subject: [Python-checkins] bpo-45402: Fix test_tools.test_sundry() (GH-28786) Message-ID: https://github.com/python/cpython/commit/ff8859d965815e8b5af346bd90299cfa5568c855 commit: ff8859d965815e8b5af346bd90299cfa5568c855 branch: main author: Victor Stinner committer: vstinner date: 2021-10-07T21:19:13+02:00 summary: bpo-45402: Fix test_tools.test_sundry() (GH-28786) Fix test_tools.test_sundry() when Python is built out of tree: fix how the freeze_modules.py tool locates the _freeze_module program. files: A Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst M Tools/scripts/freeze_modules.py diff --git a/Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst b/Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst new file mode 100644 index 0000000000000..d8c655306584c --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-07-13-43-01.bpo-45402.jlQvep.rst @@ -0,0 +1,3 @@ +Fix test_tools.test_sundry() when Python is built out of tree: fix how +the freeze_modules.py tool locates the _freeze_module program. +Patch by Victor Stinner. diff --git a/Tools/scripts/freeze_modules.py b/Tools/scripts/freeze_modules.py index 36e284100ed52..5c7eee4295289 100644 --- a/Tools/scripts/freeze_modules.py +++ b/Tools/scripts/freeze_modules.py @@ -29,9 +29,10 @@ if sys.platform != "win32": TOOL = os.path.join(ROOT_DIR, 'Programs', '_freeze_module') if not os.path.isfile(TOOL): - # When building out of the source tree, get the tool from the current - # directory - TOOL = os.path.join('Programs', '_freeze_module') + # When building out of the source tree, get the tool from directory + # of the Python executable + TOOL = os.path.dirname(sys.executable) + TOOL = os.path.join(TOOL, 'Programs', '_freeze_module') TOOL = os.path.abspath(TOOL) if not os.path.isfile(TOOL): sys.exit("ERROR: missing _freeze_module") From webhook-mailer at python.org Thu Oct 7 15:22:34 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 07 Oct 2021 19:22:34 -0000 Subject: [Python-checkins] bpo-45403: Fix test_sys.test_stdlib_dir() (GH-28785) Message-ID: https://github.com/python/cpython/commit/768aaf6c433e6a13b82c7bdebd0062c7472c1fc7 commit: 768aaf6c433e6a13b82c7bdebd0062c7472c1fc7 branch: main author: Victor Stinner committer: vstinner date: 2021-10-07T21:22:28+02:00 summary: bpo-45403: Fix test_sys.test_stdlib_dir() (GH-28785) Fix test_sys.test_stdlib_dir() when Python is built outside the source tree: compare normalized paths. files: A Misc/NEWS.d/next/Tests/2021-10-07-13-27-12.bpo-45403.7QiDvw.rst M Lib/test/test_sys.py diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index cf708407c2903..6720cd622d7c5 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1001,8 +1001,8 @@ def test_stdlib_dir(self): if marker and not os.path.exists(marker): marker = None expected = os.path.dirname(marker) if marker else None - actual = sys._stdlib_dir - self.assertEqual(actual, expected) + self.assertEqual(os.path.normpath(sys._stdlib_dir), + os.path.normpath(expected)) @test.support.cpython_only diff --git a/Misc/NEWS.d/next/Tests/2021-10-07-13-27-12.bpo-45403.7QiDvw.rst b/Misc/NEWS.d/next/Tests/2021-10-07-13-27-12.bpo-45403.7QiDvw.rst new file mode 100644 index 0000000000000..e4d1709960980 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-07-13-27-12.bpo-45403.7QiDvw.rst @@ -0,0 +1,2 @@ +Fix test_sys.test_stdlib_dir() when Python is built outside the source tree: +compare normalized paths. Patch by Victor Stinner. From webhook-mailer at python.org Thu Oct 7 15:48:26 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 19:48:26 -0000 Subject: [Python-checkins] bpo-16379: Fix SQLite version checks in test_module_constants() (GH-28809) Message-ID: https://github.com/python/cpython/commit/8deb7afbaaaad847656842375119f8dbef8aea54 commit: 8deb7afbaaaad847656842375119f8dbef8aea54 branch: main author: Erlend Egeberg Aasland committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T12:48:13-07:00 summary: bpo-16379: Fix SQLite version checks in test_module_constants() (GH-28809) Automerge-Triggered-By: GH:pablogsal files: M Lib/sqlite3/test/test_dbapi.py diff --git a/Lib/sqlite3/test/test_dbapi.py b/Lib/sqlite3/test/test_dbapi.py index c21688d2a774c..d82543663d18b 100644 --- a/Lib/sqlite3/test/test_dbapi.py +++ b/Lib/sqlite3/test/test_dbapi.py @@ -168,9 +168,9 @@ def test_module_constants(self): "SQLITE_TRANSACTION", "SQLITE_UPDATE", ] - if sqlite.version_info >= (3, 7, 17): + if sqlite.sqlite_version_info >= (3, 7, 17): consts += ["SQLITE_NOTICE", "SQLITE_WARNING"] - if sqlite.version_info >= (3, 8, 3): + if sqlite.sqlite_version_info >= (3, 8, 3): consts.append("SQLITE_RECURSIVE") consts += ["PARSE_DECLTYPES", "PARSE_COLNAMES"] for const in consts: From webhook-mailer at python.org Thu Oct 7 16:26:22 2021 From: webhook-mailer at python.org (zooba) Date: Thu, 07 Oct 2021 20:26:22 -0000 Subject: [Python-checkins] bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) Message-ID: https://github.com/python/cpython/commit/6811fdaec825bd6ab64e358a4b480108f5634d2d commit: 6811fdaec825bd6ab64e358a4b480108f5634d2d branch: main author: Steve Dower committer: zooba date: 2021-10-07T21:26:12+01:00 summary: bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) files: A Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst M Lib/test/test_venv.py M Lib/venv/__init__.py diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py index 098ba17af5975..94d626598bac3 100644 --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -150,14 +150,20 @@ def test_prompt(self): def test_upgrade_dependencies(self): builder = venv.EnvBuilder() bin_path = 'Scripts' if sys.platform == 'win32' else 'bin' - python_exe = 'python.exe' if sys.platform == 'win32' else 'python' + python_exe = os.path.split(sys.executable)[1] with tempfile.TemporaryDirectory() as fake_env_dir: + expect_exe = os.path.normcase( + os.path.join(fake_env_dir, bin_path, python_exe) + ) + if sys.platform == 'win32': + expect_exe = os.path.normcase(os.path.realpath(expect_exe)) def pip_cmd_checker(cmd): + cmd[0] = os.path.normcase(cmd[0]) self.assertEqual( cmd, [ - os.path.join(fake_env_dir, bin_path, python_exe), + expect_exe, '-m', 'pip', 'install', diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py index b007e25f7ea2c..6f1af294ae63e 100644 --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -142,6 +142,20 @@ def create_if_needed(d): context.bin_name = binname context.env_exe = os.path.join(binpath, exename) create_if_needed(binpath) + # Assign and update the command to use when launching the newly created + # environment, in case it isn't simply the executable script (e.g. bpo-45337) + context.env_exec_cmd = context.env_exe + if sys.platform == 'win32': + # bpo-45337: Fix up env_exec_cmd to account for file system redirections. + # Some redirects only apply to CreateFile and not CreateProcess + real_env_exe = os.path.realpath(context.env_exe) + if os.path.normcase(real_env_exe) != os.path.normcase(context.env_exe): + logger.warning('Actual environment location may have moved due to ' + 'redirects, links or junctions.\n' + ' Requested location: "%s"\n' + ' Actual location: "%s"', + context.env_exe, real_env_exe) + context.env_exec_cmd = real_env_exe return context def create_configuration(self, context): @@ -294,8 +308,8 @@ def _setup_pip(self, context): # We run ensurepip in isolated mode to avoid side effects from # environment vars, the current directory and anything else # intended for the global Python environment - cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade', - '--default-pip'] + cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade', + '--default-pip'] subprocess.check_output(cmd, stderr=subprocess.STDOUT) def setup_scripts(self, context): @@ -395,11 +409,7 @@ def upgrade_dependencies(self, context): logger.debug( f'Upgrading {CORE_VENV_DEPS} packages in {context.bin_path}' ) - if sys.platform == 'win32': - python_exe = os.path.join(context.bin_path, 'python.exe') - else: - python_exe = os.path.join(context.bin_path, 'python') - cmd = [python_exe, '-m', 'pip', 'install', '--upgrade'] + cmd = [context.env_exec_cmd, '-m', 'pip', 'install', '--upgrade'] cmd.extend(CORE_VENV_DEPS) subprocess.check_call(cmd) diff --git a/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst new file mode 100644 index 0000000000000..007ee87195d6e --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst @@ -0,0 +1,4 @@ +venv now warns when the created environment may need to be accessed at a +different path, due to redirections, links or junctions. It also now +correctly installs or upgrades components when the alternate path is +required. From webhook-mailer at python.org Thu Oct 7 17:33:15 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 07 Oct 2021 21:33:15 -0000 Subject: [Python-checkins] bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812) Message-ID: https://github.com/python/cpython/commit/0219017df7ec41839fd0d56a3076b5f09c58d313 commit: 0219017df7ec41839fd0d56a3076b5f09c58d313 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-07T22:33:05+01:00 summary: bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst M Lib/test/test_ast.py M Lib/test/test_exceptions.py M Parser/pegen.c diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 1f4257b1191ba..e630677f79684 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -1075,6 +1075,14 @@ def test_literal_eval_malformed_lineno(self): with self.assertRaisesRegex(ValueError, msg): ast.literal_eval(node) + def test_literal_eval_syntax_errors(self): + msg = "unexpected character after line continuation character" + with self.assertRaisesRegex(SyntaxError, msg): + ast.literal_eval(r''' + \ + (\ + \ ''') + def test_bad_integer(self): # issue13436: Bad error message with invalid numeric values body = [ast.ImportFrom(module='time', diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 85dc3c0f22081..02489a2bd9215 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -223,7 +223,7 @@ def testSyntaxErrorOffset(self): check('x = "a', 1, 5) check('lambda x: x = 2', 1, 1) check('f{a + b + c}', 1, 1) - check('[file for str(file) in []\n])', 2, 2) + check('[file for str(file) in []\n])', 1, 11) check('a = ? hello ? ? world ?', 1, 5) check('[\nfile\nfor str(file)\nin\n[]\n]', 3, 5) check('[file for\n str(file) in []]', 2, 2) diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst new file mode 100644 index 0000000000000..e4d4db9cb9536 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst @@ -0,0 +1,2 @@ +Fix a crash in the parser when reporting tokenizer errors that occur at the +same time unclosed parentheses are detected. Patch by Pablo Galindo. diff --git a/Parser/pegen.c b/Parser/pegen.c index 7e2d37caae37a..a9896356e5b58 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -1342,13 +1342,16 @@ _PyPegen_run_parser(Parser *p) { void *res = _PyPegen_parse(p); if (res == NULL) { + if (PyErr_Occurred() && !PyErr_ExceptionMatches(PyExc_SyntaxError)) { + return NULL; + } Token *last_token = p->tokens[p->fill - 1]; reset_parser_state(p); _PyPegen_parse(p); if (PyErr_Occurred()) { // Prioritize tokenizer errors to custom syntax errors raised // on the second phase only if the errors come from the parser. - if (p->tok->done != E_ERROR && PyErr_ExceptionMatches(PyExc_SyntaxError)) { + if (p->tok->done == E_DONE && PyErr_ExceptionMatches(PyExc_SyntaxError)) { _PyPegen_check_tokenizer_errors(p); } return NULL; From webhook-mailer at python.org Thu Oct 7 18:47:03 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 07 Oct 2021 22:47:03 -0000 Subject: [Python-checkins] bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) Message-ID: https://github.com/python/cpython/commit/392a89835371baa0fc4bf79ae479abb80661f57d commit: 392a89835371baa0fc4bf79ae479abb80661f57d branch: main author: Matthias Reichl committer: vstinner date: 2021-10-08T00:46:49+02:00 summary: bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) Prevent use-after-free of running loop holder via cache. files: A Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst M Modules/_asynciomodule.c diff --git a/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst new file mode 100644 index 0000000000000..4cd949fe1ed5d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst @@ -0,0 +1 @@ +Prevent use-after-free in asyncio. Make sure the cached running loop holder gets cleared on dealloc to prevent use-after-free in get_running_loop \ No newline at end of file diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index ecc73d1ca8bf0..56079b0277d1a 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3239,6 +3239,9 @@ new_running_loop_holder(PyObject *loop) static void PyRunningLoopHolder_tp_dealloc(PyRunningLoopHolder *rl) { + if (cached_running_holder == (PyObject *)rl) { + cached_running_holder = NULL; + } Py_CLEAR(rl->rl_loop); PyObject_Free(rl); } From webhook-mailer at python.org Thu Oct 7 18:55:13 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 22:55:13 -0000 Subject: [Python-checkins] bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) Message-ID: https://github.com/python/cpython/commit/eabca6e593269301a0b7a36c4dc3525f04f5bb36 commit: eabca6e593269301a0b7a36c4dc3525f04f5bb36 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T15:55:05-07:00 summary: bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) (cherry picked from commit 6811fdaec825bd6ab64e358a4b480108f5634d2d) Co-authored-by: Steve Dower files: A Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst M Lib/test/test_venv.py M Lib/venv/__init__.py diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py index 098ba17af5975..94d626598bac3 100644 --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -150,14 +150,20 @@ def test_prompt(self): def test_upgrade_dependencies(self): builder = venv.EnvBuilder() bin_path = 'Scripts' if sys.platform == 'win32' else 'bin' - python_exe = 'python.exe' if sys.platform == 'win32' else 'python' + python_exe = os.path.split(sys.executable)[1] with tempfile.TemporaryDirectory() as fake_env_dir: + expect_exe = os.path.normcase( + os.path.join(fake_env_dir, bin_path, python_exe) + ) + if sys.platform == 'win32': + expect_exe = os.path.normcase(os.path.realpath(expect_exe)) def pip_cmd_checker(cmd): + cmd[0] = os.path.normcase(cmd[0]) self.assertEqual( cmd, [ - os.path.join(fake_env_dir, bin_path, python_exe), + expect_exe, '-m', 'pip', 'install', diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py index 8009deb3ea700..ce1f5d710ad76 100644 --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -142,6 +142,20 @@ def create_if_needed(d): context.bin_name = binname context.env_exe = os.path.join(binpath, exename) create_if_needed(binpath) + # Assign and update the command to use when launching the newly created + # environment, in case it isn't simply the executable script (e.g. bpo-45337) + context.env_exec_cmd = context.env_exe + if sys.platform == 'win32': + # bpo-45337: Fix up env_exec_cmd to account for file system redirections. + # Some redirects only apply to CreateFile and not CreateProcess + real_env_exe = os.path.realpath(context.env_exe) + if os.path.normcase(real_env_exe) != os.path.normcase(context.env_exe): + logger.warning('Actual environment location may have moved due to ' + 'redirects, links or junctions.\n' + ' Requested location: "%s"\n' + ' Actual location: "%s"', + context.env_exe, real_env_exe) + context.env_exec_cmd = real_env_exe return context def create_configuration(self, context): @@ -293,8 +307,8 @@ def _setup_pip(self, context): # We run ensurepip in isolated mode to avoid side effects from # environment vars, the current directory and anything else # intended for the global Python environment - cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade', - '--default-pip'] + cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade', + '--default-pip'] subprocess.check_output(cmd, stderr=subprocess.STDOUT) def setup_scripts(self, context): @@ -394,11 +408,7 @@ def upgrade_dependencies(self, context): logger.debug( f'Upgrading {CORE_VENV_DEPS} packages in {context.bin_path}' ) - if sys.platform == 'win32': - python_exe = os.path.join(context.bin_path, 'python.exe') - else: - python_exe = os.path.join(context.bin_path, 'python') - cmd = [python_exe, '-m', 'pip', 'install', '--upgrade'] + cmd = [context.env_exec_cmd, '-m', 'pip', 'install', '--upgrade'] cmd.extend(CORE_VENV_DEPS) subprocess.check_call(cmd) diff --git a/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst new file mode 100644 index 0000000000000..007ee87195d6e --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst @@ -0,0 +1,4 @@ +venv now warns when the created environment may need to be accessed at a +different path, due to redirections, links or junctions. It also now +correctly installs or upgrades components when the alternate path is +required. From webhook-mailer at python.org Thu Oct 7 18:55:20 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 22:55:20 -0000 Subject: [Python-checkins] bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) Message-ID: https://github.com/python/cpython/commit/06935bd68e3d89a4cc3f08c1d15eaa651b79a523 commit: 06935bd68e3d89a4cc3f08c1d15eaa651b79a523 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T15:55:15-07:00 summary: bpo-45337: Use the realpath of the new executable when creating a venv on Windows (GH-28663) (cherry picked from commit 6811fdaec825bd6ab64e358a4b480108f5634d2d) Co-authored-by: Steve Dower files: A Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst M Lib/test/test_venv.py M Lib/venv/__init__.py diff --git a/Lib/test/test_venv.py b/Lib/test/test_venv.py index ca003d55d7c89..480cb29f35a6a 100644 --- a/Lib/test/test_venv.py +++ b/Lib/test/test_venv.py @@ -151,14 +151,20 @@ def test_prompt(self): def test_upgrade_dependencies(self): builder = venv.EnvBuilder() bin_path = 'Scripts' if sys.platform == 'win32' else 'bin' - python_exe = 'python.exe' if sys.platform == 'win32' else 'python' + python_exe = os.path.split(sys.executable)[1] with tempfile.TemporaryDirectory() as fake_env_dir: + expect_exe = os.path.normcase( + os.path.join(fake_env_dir, bin_path, python_exe) + ) + if sys.platform == 'win32': + expect_exe = os.path.normcase(os.path.realpath(expect_exe)) def pip_cmd_checker(cmd): + cmd[0] = os.path.normcase(cmd[0]) self.assertEqual( cmd, [ - os.path.join(fake_env_dir, bin_path, python_exe), + expect_exe, '-m', 'pip', 'install', diff --git a/Lib/venv/__init__.py b/Lib/venv/__init__.py index 8009deb3ea700..ce1f5d710ad76 100644 --- a/Lib/venv/__init__.py +++ b/Lib/venv/__init__.py @@ -142,6 +142,20 @@ def create_if_needed(d): context.bin_name = binname context.env_exe = os.path.join(binpath, exename) create_if_needed(binpath) + # Assign and update the command to use when launching the newly created + # environment, in case it isn't simply the executable script (e.g. bpo-45337) + context.env_exec_cmd = context.env_exe + if sys.platform == 'win32': + # bpo-45337: Fix up env_exec_cmd to account for file system redirections. + # Some redirects only apply to CreateFile and not CreateProcess + real_env_exe = os.path.realpath(context.env_exe) + if os.path.normcase(real_env_exe) != os.path.normcase(context.env_exe): + logger.warning('Actual environment location may have moved due to ' + 'redirects, links or junctions.\n' + ' Requested location: "%s"\n' + ' Actual location: "%s"', + context.env_exe, real_env_exe) + context.env_exec_cmd = real_env_exe return context def create_configuration(self, context): @@ -293,8 +307,8 @@ def _setup_pip(self, context): # We run ensurepip in isolated mode to avoid side effects from # environment vars, the current directory and anything else # intended for the global Python environment - cmd = [context.env_exe, '-Im', 'ensurepip', '--upgrade', - '--default-pip'] + cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade', + '--default-pip'] subprocess.check_output(cmd, stderr=subprocess.STDOUT) def setup_scripts(self, context): @@ -394,11 +408,7 @@ def upgrade_dependencies(self, context): logger.debug( f'Upgrading {CORE_VENV_DEPS} packages in {context.bin_path}' ) - if sys.platform == 'win32': - python_exe = os.path.join(context.bin_path, 'python.exe') - else: - python_exe = os.path.join(context.bin_path, 'python') - cmd = [python_exe, '-m', 'pip', 'install', '--upgrade'] + cmd = [context.env_exec_cmd, '-m', 'pip', 'install', '--upgrade'] cmd.extend(CORE_VENV_DEPS) subprocess.check_call(cmd) diff --git a/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst new file mode 100644 index 0000000000000..007ee87195d6e --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-09-30-23-17-27.bpo-45337.qg7U_h.rst @@ -0,0 +1,4 @@ +venv now warns when the created environment may need to be accessed at a +different path, due to redirections, links or junctions. It also now +correctly installs or upgrades components when the alternate path is +required. From webhook-mailer at python.org Thu Oct 7 19:14:12 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 07 Oct 2021 23:14:12 -0000 Subject: [Python-checkins] bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) Message-ID: https://github.com/python/cpython/commit/87f0156a229e4cda92ad8e50645c5a71030caf7c commit: 87f0156a229e4cda92ad8e50645c5a71030caf7c branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-07T16:14:04-07:00 summary: bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) Prevent use-after-free of running loop holder via cache. (cherry picked from commit 392a89835371baa0fc4bf79ae479abb80661f57d) Co-authored-by: Matthias Reichl files: A Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst M Modules/_asynciomodule.c diff --git a/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst new file mode 100644 index 0000000000000..4cd949fe1ed5d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst @@ -0,0 +1 @@ +Prevent use-after-free in asyncio. Make sure the cached running loop holder gets cleared on dealloc to prevent use-after-free in get_running_loop \ No newline at end of file diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index b615c48c43182..4457d7bd49e23 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3258,6 +3258,9 @@ new_running_loop_holder(PyObject *loop) static void PyRunningLoopHolder_tp_dealloc(PyRunningLoopHolder *rl) { + if (cached_running_holder == (PyObject *)rl) { + cached_running_holder = NULL; + } Py_CLEAR(rl->rl_loop); PyObject_Free(rl); } From webhook-mailer at python.org Thu Oct 7 19:50:19 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 07 Oct 2021 23:50:19 -0000 Subject: [Python-checkins] [3.10] bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812). (GH-28813) Message-ID: https://github.com/python/cpython/commit/4ce55a2353e07962280181df40af0135aef1cf51 commit: 4ce55a2353e07962280181df40af0135aef1cf51 branch: 3.10 author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-08T00:50:10+01:00 summary: [3.10] bpo-45408: Don't override previous tokenizer errors in the second parser pass (GH-28812). (GH-28813) (cherry picked from commit 0219017df7ec41839fd0d56a3076b5f09c58d313) Co-authored-by: Pablo Galindo Salgado files: A Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst M Lib/test/test_ast.py M Lib/test/test_exceptions.py M Parser/pegen.c diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index 326f3ab2beb1d..39fc7e9673816 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -1044,6 +1044,14 @@ def test_literal_eval_malformed_lineno(self): with self.assertRaisesRegex(ValueError, msg): ast.literal_eval(node) + def test_literal_eval_syntax_errors(self): + msg = "unexpected character after line continuation character" + with self.assertRaisesRegex(SyntaxError, msg): + ast.literal_eval(r''' + \ + (\ + \ ''') + def test_bad_integer(self): # issue13436: Bad error message with invalid numeric values body = [ast.ImportFrom(module='time', diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index d04e5f5573ca9..4213dabfd8e71 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -223,7 +223,7 @@ def testSyntaxErrorOffset(self): check('x = "a', 1, 5) check('lambda x: x = 2', 1, 1) check('f{a + b + c}', 1, 2) - check('[file for str(file) in []\n])', 2, 2) + check('[file for str(file) in []\n])', 1, 11) check('a = ? hello ? ? world ?', 1, 5) check('[\nfile\nfor str(file)\nin\n[]\n]', 3, 5) check('[file for\n str(file) in []]', 2, 2) diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst new file mode 100644 index 0000000000000..e4d4db9cb9536 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-21-26-44.bpo-45408.qUqzcd.rst @@ -0,0 +1,2 @@ +Fix a crash in the parser when reporting tokenizer errors that occur at the +same time unclosed parentheses are detected. Patch by Pablo Galindo. diff --git a/Parser/pegen.c b/Parser/pegen.c index e20e926136828..1bb975d684a0d 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -1321,13 +1321,16 @@ _PyPegen_run_parser(Parser *p) { void *res = _PyPegen_parse(p); if (res == NULL) { + if (PyErr_Occurred() && !PyErr_ExceptionMatches(PyExc_SyntaxError)) { + return NULL; + } Token *last_token = p->tokens[p->fill - 1]; reset_parser_state(p); _PyPegen_parse(p); if (PyErr_Occurred()) { // Prioritize tokenizer errors to custom syntax errors raised // on the second phase only if the errors come from the parser. - if (p->tok->done != E_ERROR && PyErr_ExceptionMatches(PyExc_SyntaxError)) { + if (p->tok->done == E_DONE && PyErr_ExceptionMatches(PyExc_SyntaxError)) { _PyPegen_check_tokenizer_errors(p); } return NULL; From webhook-mailer at python.org Fri Oct 8 02:45:19 2021 From: webhook-mailer at python.org (corona10) Date: Fri, 08 Oct 2021 06:45:19 -0000 Subject: [Python-checkins] bpo-45407: Remove outdated XXX comment from Struct___init___impl (GH-28805) Message-ID: https://github.com/python/cpython/commit/9f7a94fd66e05ae040a67e32c397091fe5939ced commit: 9f7a94fd66e05ae040a67e32c397091fe5939ced branch: main author: Jeong YunWon <69878+youknowone at users.noreply.github.com> committer: corona10 date: 2021-10-08T15:45:11+09:00 summary: bpo-45407: Remove outdated XXX comment from Struct___init___impl (GH-28805) files: M Modules/_struct.c diff --git a/Modules/_struct.c b/Modules/_struct.c index 872c30d659d82..69de080f4388c 100644 --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -1475,7 +1475,6 @@ Struct___init___impl(PyStructObject *self, PyObject *format) if (format == NULL) return -1; } - /* XXX support buffer interface, too */ else { Py_INCREF(format); } From webhook-mailer at python.org Fri Oct 8 04:55:46 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 08 Oct 2021 08:55:46 -0000 Subject: [Python-checkins] bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) (GH-28816) Message-ID: https://github.com/python/cpython/commit/6846d6712a0894f8e1a91716c11dd79f42864216 commit: 6846d6712a0894f8e1a91716c11dd79f42864216 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vstinner date: 2021-10-08T10:55:41+02:00 summary: bpo-45262, asyncio: Fix cache of the running loop holder (GH-28796) (GH-28816) Prevent use-after-free of running loop holder via cache. (cherry picked from commit 392a89835371baa0fc4bf79ae479abb80661f57d) Co-authored-by: Matthias Reichl files: A Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst M Modules/_asynciomodule.c diff --git a/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst new file mode 100644 index 0000000000000..4cd949fe1ed5d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-14-04-10.bpo-45262.HqF71Z.rst @@ -0,0 +1 @@ +Prevent use-after-free in asyncio. Make sure the cached running loop holder gets cleared on dealloc to prevent use-after-free in get_running_loop \ No newline at end of file diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index ecc73d1ca8bf0..56079b0277d1a 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -3239,6 +3239,9 @@ new_running_loop_holder(PyObject *loop) static void PyRunningLoopHolder_tp_dealloc(PyRunningLoopHolder *rl) { + if (cached_running_holder == (PyObject *)rl) { + cached_running_holder = NULL; + } Py_CLEAR(rl->rl_loop); PyObject_Free(rl); } From webhook-mailer at python.org Fri Oct 8 11:14:50 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 08 Oct 2021 15:14:50 -0000 Subject: [Python-checkins] bpo-45410: libregrtest -jN writes stderr into stdout (GH-28819) Message-ID: https://github.com/python/cpython/commit/b108db63e02797a795840152b82cab9792fd3ed2 commit: b108db63e02797a795840152b82cab9792fd3ed2 branch: main author: Victor Stinner committer: vstinner date: 2021-10-08T17:14:37+02:00 summary: bpo-45410: libregrtest -jN writes stderr into stdout (GH-28819) When libregrtest spawns a worker process, stderr is now written into stdout to keep messages order. Use a single pipe for stdout and stderr, rather than two pipes. Previously, messages were out of order which made analysis of buildbot logs harder files: A Misc/NEWS.d/next/Tests/2021-10-08-14-03-20.bpo-45410.Ex9xe2.rst M Lib/test/libregrtest/runtest_mp.py diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py index c83e44aed0514..dea5992feb7b4 100644 --- a/Lib/test/libregrtest/runtest_mp.py +++ b/Lib/test/libregrtest/runtest_mp.py @@ -70,7 +70,9 @@ def run_test_in_subprocess(testname: str, ns: Namespace) -> subprocess.Popen: kw['start_new_session'] = True return subprocess.Popen(cmd, stdout=subprocess.PIPE, - stderr=subprocess.PIPE, + # bpo-45410: Write stderr into stdout to keep + # messages order + stderr=subprocess.STDOUT, universal_newlines=True, close_fds=(os.name != 'nt'), cwd=os_helper.SAVEDCWD, @@ -114,8 +116,8 @@ def stop(self): class MultiprocessResult(NamedTuple): result: TestResult + # bpo-45410: stderr is written into stdout to keep messages order stdout: str - stderr: str error_msg: str @@ -195,11 +197,10 @@ def mp_result_error( self, test_result: TestResult, stdout: str = '', - stderr: str = '', err_msg=None ) -> MultiprocessResult: test_result.duration_sec = time.monotonic() - self.start_time - return MultiprocessResult(test_result, stdout, stderr, err_msg) + return MultiprocessResult(test_result, stdout, err_msg) def _run_process(self, test_name: str) -> tuple[int, str, str]: self.start_time = time.monotonic() @@ -223,13 +224,14 @@ def _run_process(self, test_name: str) -> tuple[int, str, str]: raise ExitThread try: - stdout, stderr = popen.communicate(timeout=self.timeout) + # bpo-45410: stderr is written into stdout + stdout, _ = popen.communicate(timeout=self.timeout) retcode = popen.returncode assert retcode is not None except subprocess.TimeoutExpired: if self._stopped: - # kill() has been called: communicate() fails - # on reading closed stdout/stderr + # kill() has been called: communicate() fails on reading + # closed stdout raise ExitThread # On timeout, kill the process @@ -238,20 +240,19 @@ def _run_process(self, test_name: str) -> tuple[int, str, str]: # None means TIMEOUT for the caller retcode = None # bpo-38207: Don't attempt to call communicate() again: on it - # can hang until all child processes using stdout and stderr + # can hang until all child processes using stdout # pipes completes. - stdout = stderr = '' + stdout = '' except OSError: if self._stopped: # kill() has been called: communicate() fails - # on reading closed stdout/stderr + # on reading closed stdout raise ExitThread raise else: stdout = stdout.strip() - stderr = stderr.rstrip() - return (retcode, stdout, stderr) + return (retcode, stdout) except: self._kill() raise @@ -261,10 +262,10 @@ def _run_process(self, test_name: str) -> tuple[int, str, str]: self.current_test_name = None def _runtest(self, test_name: str) -> MultiprocessResult: - retcode, stdout, stderr = self._run_process(test_name) + retcode, stdout = self._run_process(test_name) if retcode is None: - return self.mp_result_error(Timeout(test_name), stdout, stderr) + return self.mp_result_error(Timeout(test_name), stdout) err_msg = None if retcode != 0: @@ -282,10 +283,9 @@ def _runtest(self, test_name: str) -> MultiprocessResult: err_msg = "Failed to parse worker JSON: %s" % exc if err_msg is not None: - return self.mp_result_error(ChildError(test_name), - stdout, stderr, err_msg) + return self.mp_result_error(ChildError(test_name), stdout, err_msg) - return MultiprocessResult(result, stdout, stderr, err_msg) + return MultiprocessResult(result, stdout, err_msg) def run(self) -> None: while not self._stopped: @@ -309,10 +309,8 @@ def run(self) -> None: def _wait_completed(self) -> None: popen = self._popen - # stdout and stderr must be closed to ensure that communicate() - # does not hang + # stdout must be closed to ensure that communicate() does not hang popen.stdout.close() - popen.stderr.close() try: popen.wait(JOIN_TIMEOUT) @@ -449,8 +447,6 @@ def _process_result(self, item: QueueOutput) -> bool: if mp_result.stdout: print(mp_result.stdout, flush=True) - if mp_result.stderr and not self.ns.pgo: - print(mp_result.stderr, file=sys.stderr, flush=True) if must_stop(mp_result.result, self.ns): return True diff --git a/Misc/NEWS.d/next/Tests/2021-10-08-14-03-20.bpo-45410.Ex9xe2.rst b/Misc/NEWS.d/next/Tests/2021-10-08-14-03-20.bpo-45410.Ex9xe2.rst new file mode 100644 index 0000000000000..5f7b8bf1d6c66 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-08-14-03-20.bpo-45410.Ex9xe2.rst @@ -0,0 +1,4 @@ +When libregrtest spawns a worker process, stderr is now written into stdout +to keep messages order. Use a single pipe for stdout and stderr, rather than +two pipes. Previously, messages were out of order which made analysis of +buildbot logs harder Patch by Victor Stinner. From webhook-mailer at python.org Fri Oct 8 23:33:46 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Sat, 09 Oct 2021 03:33:46 -0000 Subject: [Python-checkins] Replace usage of List[...] with list[...] in typing docs (GH-28821) Message-ID: https://github.com/python/cpython/commit/a98b273ce42f33d04c8b85b8d574c47adf11dd2a commit: a98b273ce42f33d04c8b85b8d574c47adf11dd2a branch: main author: Micael Jarniac committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-09T11:33:37+08:00 summary: Replace usage of List[...] with list[...] in typing docs (GH-28821) The ``List[...]`` form is deprecated since 3.9. files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 13760c19214ee..bc6130ace4a4b 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -975,16 +975,16 @@ These can be used as types in annotations using ``[]``, each having a unique syn For example:: - def is_str_list(val: List[object]) -> TypeGuard[List[str]]: + def is_str_list(val: list[object]) -> TypeGuard[list[str]]: '''Determines whether all objects in the list are strings''' return all(isinstance(x, str) for x in val) - def func1(val: List[object]): + def func1(val: list[object]): if is_str_list(val): - # Type of ``val`` is narrowed to ``List[str]``. + # Type of ``val`` is narrowed to ``list[str]``. print(" ".join(val)) else: - # Type of ``val`` remains as ``List[object]``. + # Type of ``val`` remains as ``list[object]``. print("Not a list of strings!") If ``is_str_list`` is a class or instance method, then the type in @@ -999,8 +999,8 @@ These can be used as types in annotations using ``[]``, each having a unique syn ``TypeB`` need not be a narrower form of ``TypeA`` -- it can even be a wider form. The main reason is to allow for things like - narrowing ``List[object]`` to ``List[str]`` even though the latter - is not a subtype of the former, since ``List`` is invariant. + narrowing ``list[object]`` to ``list[str]`` even though the latter + is not a subtype of the former, since ``list`` is invariant. The responsibility of writing type-safe type guards is left to the user. ``TypeGuard`` also works with type variables. For more information, see @@ -2065,8 +2065,8 @@ Introspection helpers .. class:: ForwardRef A class used for internal typing representation of string forward references. - For example, ``List["SomeClass"]`` is implicitly transformed into - ``List[ForwardRef("SomeClass")]``. This class should not be instantiated by + For example, ``list["SomeClass"]`` is implicitly transformed into + ``list[ForwardRef("SomeClass")]``. This class should not be instantiated by a user, but may be used by introspection tools. .. note:: From webhook-mailer at python.org Sat Oct 9 03:36:59 2021 From: webhook-mailer at python.org (JulienPalard) Date: Sat, 09 Oct 2021 07:36:59 -0000 Subject: [Python-checkins] bpo-10716: Migrating pydoc to html5. (GH-28651) Message-ID: https://github.com/python/cpython/commit/c91b6f57f3f75b482e4a9d30ad2afe37892a8ceb commit: c91b6f57f3f75b482e4a9d30ad2afe37892a8ceb branch: main author: Julien Palard committer: JulienPalard date: 2021-10-09T09:36:50+02:00 summary: bpo-10716: Migrating pydoc to html5. (GH-28651) files: A Misc/NEWS.d/next/Library/2021-10-08-04-11-55.bpo-10716.QSRVK2.rst M Lib/cgitb.py M Lib/pydoc.py M Lib/pydoc_data/_pydoc.css M Lib/test/test_docxmlrpc.py M Lib/test/test_pydoc.py M Lib/xmlrpc/server.py diff --git a/Lib/cgitb.py b/Lib/cgitb.py index 17ddda376884d..ec156843099d3 100644 --- a/Lib/cgitb.py +++ b/Lib/cgitb.py @@ -31,6 +31,7 @@ import time import tokenize import traceback +from html import escape as html_escape def reset(): """Return a string that resets the CGI and browser to a known state.""" @@ -105,10 +106,16 @@ def html(einfo, context=5): etype = etype.__name__ pyver = 'Python ' + sys.version.split()[0] + ': ' + sys.executable date = time.ctime(time.time()) - head = '' + pydoc.html.heading( - '%s' % - strong(pydoc.html.escape(str(etype))), - '#ffffff', '#6622aa', pyver + '
' + date) + ''' + head = f''' + + + + + +
 
+ 
+{html_escape(str(etype))}		 +{pyver}
{date}

A problem occurred in a Python script. Here is the sequence of function calls leading up to the error, in the order they occurred.

''' diff --git a/Lib/pydoc.py b/Lib/pydoc.py index 34a608760338e..3a2ff218f8319 100755 --- a/Lib/pydoc.py +++ b/Lib/pydoc.py @@ -542,7 +542,7 @@ def repr_string(self, x, level): # needed to make any special characters, so show a raw string. return 'r' + testrepr[0] + self.escape(test) + testrepr[0] return re.sub(r'((\\[\\abfnrtv\'"]|\\[0-9]..|\\x..|\\u....)+)', - r'\1', + r'\1', self.escape(testrepr)) repr_str = repr_string @@ -567,49 +567,48 @@ class HTMLDoc(Doc): def page(self, title, contents): """Format an HTML page.""" return '''\ - -Python: %s - - + + + + +Python: %s + %s ''' % (title, contents) - def heading(self, title, fgcol, bgcol, extras=''): + def heading(self, title, extras=''): """Format a page heading.""" return ''' - - -
 
- 
%s		%s
- ''' % (bgcol, fgcol, title, fgcol, extras or ' ') - - def section(self, title, fgcol, bgcol, contents, width=6, + + + +
 
%s		%s
+ ''' % (title, extras or ' ') + + def section(self, title, cls, contents, width=6, prelude='', marginalia=None, gap=' '): """Format a section with a heading.""" if marginalia is None: - marginalia = '' + ' ' * width + '' + marginalia = '' + ' ' * width + '' result = '''

- - - - ''' % (bgcol, fgcol, title) +
 
-%s
+ + + ''' % (cls, title) if prelude: result = result + ''' - - -''' % (bgcol, marginalia, prelude, gap) + + +''' % (cls, marginalia, cls, prelude, gap) else: result = result + ''' -''' % (bgcol, marginalia, gap) +''' % (cls, marginalia, gap) - return result + '\n
 
%s
%s%s
%s
%s%s
%s
%s%s
%s%s%s
' % contents + return result + '\n%s' % contents def bigsection(self, title, *args): """Format a section with a big heading.""" - title = '%s' % title + title = '%s' % title return self.section(title, *args) def preformat(self, text): @@ -618,19 +617,19 @@ def preformat(self, text): return replace(text, '\n\n', '\n \n', '\n\n', '\n \n', ' ', ' ', '\n', '
\n') - def multicolumn(self, list, format, cols=4): + def multicolumn(self, list, format): """Format a list of items into a multi-column list.""" result = '' - rows = (len(list)+cols-1)//cols - for col in range(cols): - result = result + '' % (100//cols) + rows = (len(list) + 3) // 4 + for col in range(4): + result = result + '' for i in range(rows*col, rows*col+rows): if i < len(list): result = result + format(list[i]) + '
\n' result = result + '' - return '%s
' % result + return '%s
' % result - def grey(self, text): return '%s' % text + def grey(self, text): return '%s' % text def namelink(self, name, *dicts): """Make a link for an identifier, given name-to-URL mappings.""" @@ -719,14 +718,14 @@ def formattree(self, tree, modname, parent=None): for entry in tree: if type(entry) is type(()): c, bases = entry - result = result + '

' + result = result + '
' result = result + self.classlink(c, modname) if bases and bases != (parent,): parents = [] for base in bases: parents.append(self.classlink(base, modname)) result = result + '(' + ', '.join(parents) + ')' - result = result + '\n
' + result = result + '\n' elif type(entry) is type([]): result = result + '
\n%s
\n' % self.formattree( entry, modname, c) @@ -743,10 +742,10 @@ def docmodule(self, object, name=None, mod=None, *ignored): links = [] for i in range(len(parts)-1): links.append( - '%s' % + '%s' % ('.'.join(parts[:i+1]), parts[i])) linkedname = '.'.join(links + parts[-1:]) - head = '%s' % linkedname + head = '%s' % linkedname try: path = inspect.getabsfile(object) url = urllib.parse.quote(path) @@ -768,9 +767,7 @@ def docmodule(self, object, name=None, mod=None, *ignored): docloc = '
Module Reference' % locals() else: docloc = '' - result = self.heading( - head, '#ffffff', '#7799ee', - 'index
' + filelink + docloc) + result = self.heading(head, 'index
' + filelink + docloc) modules = inspect.getmembers(object, inspect.ismodule) @@ -805,7 +802,7 @@ def docmodule(self, object, name=None, mod=None, *ignored): data.append((key, value)) doc = self.markup(getdoc(object), self.preformat, fdict, cdict) - doc = doc and '%s' % doc + doc = doc and '%s' % doc result = result + '

%s

\n' % doc if hasattr(object, '__path__'): @@ -815,12 +812,12 @@ def docmodule(self, object, name=None, mod=None, *ignored): modpkgs.sort() contents = self.multicolumn(modpkgs, self.modpkglink) result = result + self.bigsection( - 'Package Contents', '#ffffff', '#aa55cc', contents) + 'Package Contents', 'pkg-content', contents) elif modules: contents = self.multicolumn( modules, lambda t: self.modulelink(t[1])) result = result + self.bigsection( - 'Modules', '#ffffff', '#aa55cc', contents) + 'Modules', 'pkg-content', contents) if classes: classlist = [value for (key, value) in classes] @@ -829,27 +826,25 @@ def docmodule(self, object, name=None, mod=None, *ignored): for key, value in classes: contents.append(self.document(value, key, name, fdict, cdict)) result = result + self.bigsection( - 'Classes', '#ffffff', '#ee77aa', ' '.join(contents)) + 'Classes', 'index', ' '.join(contents)) if funcs: contents = [] for key, value in funcs: contents.append(self.document(value, key, name, fdict, cdict)) result = result + self.bigsection( - 'Functions', '#ffffff', '#eeaa77', ' '.join(contents)) + 'Functions', 'functions', ' '.join(contents)) if data: contents = [] for key, value in data: contents.append(self.document(value, key)) result = result + self.bigsection( - 'Data', '#ffffff', '#55aa55', '
\n'.join(contents)) + 'Data', 'data', '
\n'.join(contents)) if hasattr(object, '__author__'): contents = self.markup(str(object.__author__), self.preformat) - result = result + self.bigsection( - 'Author', '#ffffff', '#7799ee', contents) + result = result + self.bigsection('Author', 'author', contents) if hasattr(object, '__credits__'): contents = self.markup(str(object.__credits__), self.preformat) - result = result + self.bigsection( - 'Credits', '#ffffff', '#7799ee', contents) + result = result + self.bigsection('Credits', 'credits', contents) return result @@ -923,7 +918,7 @@ def spilldata(msg, attrs, predicate): else: doc = self.markup(getdoc(value), self.preformat, funcs, classes, mdict) - doc = '
%s' % doc + doc = '
%s' % doc push('
%s%s
\n' % (base, doc)) push('\n') return attrs @@ -1011,9 +1006,9 @@ def spilldata(msg, attrs, predicate): if decl: doc = decl + (doc or '') doc = self.markup(doc, self.preformat, funcs, classes, mdict) - doc = doc and '%s
 ' % doc + doc = doc and '%s
 ' % doc - return self.section(title, '#000000', '#ffc8d8', contents, 3, doc) + return self.section(title, 'title', contents, 3, doc) def formatvalue(self, object): """Format an argument default value as text.""" @@ -1074,14 +1069,14 @@ def docroutine(self, object, name=None, mod=None, argspec = '(...)' decl = asyncqualifier + title + self.escape(argspec) + (note and - self.grey('%s' % note)) + self.grey('%s' % note)) if skipdocs: return '
%s
\n' % decl else: doc = self.markup( getdoc(object), self.preformat, funcs, classes, methods) - doc = doc and '
%s
' % doc + doc = doc and '
%s
' % doc return '
%s
%s
\n' % (decl, doc) def docdata(self, object, name=None, mod=None, cl=None): @@ -1093,7 +1088,7 @@ def docdata(self, object, name=None, mod=None, cl=None): push('
%s
\n' % name) doc = self.markup(getdoc(object), self.preformat) if doc: - push('
%s
\n' % doc) + push('
%s
\n' % doc) push('
\n') return ''.join(results) @@ -1118,7 +1113,7 @@ def index(self, dir, shadowed=None): modpkgs.sort() contents = self.multicolumn(modpkgs, self.modpkglink) - return self.bigsection(dir, '#ffffff', '#ee77aa', contents) + return self.bigsection(dir, 'index', contents) # -------------------------------------------- text documentation generator @@ -2446,10 +2441,12 @@ def page(self, title, contents): '' % css_path) return '''\ - -Pydoc: %s - -%s%s
%s
+ + + + +Pydoc: %s +%s%s
%s
''' % (title, css_link, html_navbar(), contents) @@ -2489,22 +2486,21 @@ def bltinlink(name): return '%s' % (name, name) heading = html.heading( - 'Index of Modules', - '#ffffff', '#7799ee') + 'Index of Modules' + ) names = [name for name in sys.builtin_module_names if name != '__main__'] contents = html.multicolumn(names, bltinlink) contents = [heading, '

' + html.bigsection( - 'Built-in Modules', '#ffffff', '#ee77aa', contents)] + 'Built-in Modules', 'index', contents)] seen = {} for dir in sys.path: contents.append(html.index(dir, seen)) contents.append( - '

pydoc by Ka-Ping Yee' - '<ping at lfw.org>') + '

pydoc by Ka-Ping Yee' + '<ping at lfw.org>

') return 'Index of Modules', ''.join(contents) def html_search(key): @@ -2529,12 +2525,12 @@ def bltinlink(name): results = [] heading = html.heading( - 'Search Results', - '#ffffff', '#7799ee') + 'Search Results', + ) for name, desc in search_result: results.append(bltinlink(name) + desc) contents = heading + html.bigsection( - 'key = %s' % key, '#ffffff', '#ee77aa', '
'.join(results)) + 'key = %s' % key, 'index', '
'.join(results)) return 'Search Results', contents def html_topics(): @@ -2544,20 +2540,20 @@ def bltinlink(name): return '%s' % (name, name) heading = html.heading( - 'INDEX', - '#ffffff', '#7799ee') + 'INDEX', + ) names = sorted(Helper.topics.keys()) contents = html.multicolumn(names, bltinlink) contents = heading + html.bigsection( - 'Topics', '#ffffff', '#ee77aa', contents) + 'Topics', 'index', contents) return 'Topics', contents def html_keywords(): """Index of keywords.""" heading = html.heading( - 'INDEX', - '#ffffff', '#7799ee') + 'INDEX', + ) names = sorted(Helper.keywords.keys()) def bltinlink(name): @@ -2565,7 +2561,7 @@ def bltinlink(name): contents = html.multicolumn(names, bltinlink) contents = heading + html.bigsection( - 'Keywords', '#ffffff', '#ee77aa', contents) + 'Keywords', 'index', contents) return 'Keywords', contents def html_topicpage(topic): @@ -2578,10 +2574,10 @@ def html_topicpage(topic): else: title = 'TOPIC' heading = html.heading( - '%s' % title, - '#ffffff', '#7799ee') + '%s' % title, + ) contents = '
%s
' % html.markup(contents) - contents = html.bigsection(topic , '#ffffff','#ee77aa', contents) + contents = html.bigsection(topic , 'index', contents) if xrefs: xrefs = sorted(xrefs.split()) @@ -2589,8 +2585,7 @@ def bltinlink(name): return '%s' % (name, name) xrefs = html.multicolumn(xrefs, bltinlink) - xrefs = html.section('Related help topics: ', - '#ffffff', '#ee77aa', xrefs) + xrefs = html.section('Related help topics: ', 'index', xrefs) return ('%s %s' % (title, topic), ''.join((heading, contents, xrefs))) @@ -2604,12 +2599,11 @@ def html_getobj(url): def html_error(url, exc): heading = html.heading( - 'Error', - '#ffffff', '#7799ee') + 'Error', + ) contents = '
'.join(html.escape(line) for line in format_exception_only(type(exc), exc)) - contents = heading + html.bigsection(url, '#ffffff', '#bb0000', - contents) + contents = heading + html.bigsection(url, 'error', contents) return "Error - %s" % url, contents def get_html_page(url): diff --git a/Lib/pydoc_data/_pydoc.css b/Lib/pydoc_data/_pydoc.css index f036ef37a5aba..a6aa2e4c1a021 100644 --- a/Lib/pydoc_data/_pydoc.css +++ b/Lib/pydoc_data/_pydoc.css @@ -4,3 +4,109 @@ Contents of this file are subject to change without notice. */ + +body { + background-color: #f0f0f8; +} + +table.heading tr { + background-color: #7799ee; +} + +.decor { + color: #ffffff; +} + +.title-decor { + background-color: #ffc8d8; + color: #000000; +} + +.pkg-content-decor { + background-color: #aa55cc; +} + +.index-decor { + background-color: #ee77aa; +} + +.functions-decor { + background-color: #eeaa77; +} + +.data-decor { + background-color: #55aa55; +} + +.author-decor { + background-color: #7799ee; +} + +.credits-decor { + background-color: #7799ee; +} + +.error-decor { + background-color: #bb0000; +} + +.grey { + color: #909090; +} + +.white { + color: #ffffff; +} + +.repr { + color: #c040c0; +} + +table.heading tr td.title { + vertical-align: bottom; +} + +table.heading tr td.extra { + vertical-align: bottom; + text-align: right; +} + +.heading-text { + font-family: helvetica, arial; +} + +.bigsection { + font-size: larger; +} + +.title { + font-size: x-large; +} + +.code { + font-family: monospace; +} + +table { + width: 100%; + border-spacing : 0; + border-collapse : collapse; + border: 0; +} + +td { + padding: 2; +} + +td.section-title { + vertical-align: bottom; +} + +td.multicolumn { + width: 25%; + vertical-align: bottom; +} + +td.singlecolumn { + width: 100%; +} diff --git a/Lib/test/test_docxmlrpc.py b/Lib/test/test_docxmlrpc.py index 7725250288311..9a06be4585502 100644 --- a/Lib/test/test_docxmlrpc.py +++ b/Lib/test/test_docxmlrpc.py @@ -90,7 +90,17 @@ def test_valid_get_response(self): response = self.client.getresponse() self.assertEqual(response.status, 200) - self.assertEqual(response.getheader("Content-type"), "text/html") + self.assertEqual(response.getheader("Content-type"), "text/html; charset=UTF-8") + + # Server raises an exception if we don't start to read the data + response.read() + + def test_get_css(self): + self.client.request("GET", "/pydoc.css") + response = self.client.getresponse() + + self.assertEqual(response.status, 200) + self.assertEqual(response.getheader("Content-type"), "text/css; charset=UTF-8") # Server raises an exception if we don't start to read the data response.read() diff --git a/Lib/test/test_pydoc.py b/Lib/test/test_pydoc.py index 25ac1fb59d4c4..0a7d72c768424 100644 --- a/Lib/test/test_pydoc.py +++ b/Lib/test/test_pydoc.py @@ -132,128 +132,70 @@ class C(builtins.object) expected_text_data_docstrings = tuple('\n | ' + s if s else '' for s in expected_data_docstrings) -expected_html_pattern = """ - - -
 
- 
test.pydoc_mod (version 1.2.3.4)		index
%s%s
-

This is a test module for test_pydoc

-

- - - -\x20\x20\x20\x20 - -
 
-Classes
       
-
builtins.object -
-
-
A -
B -
C -
-
-
-

- - - -\x20\x20\x20\x20 - - - -
 
-class A(builtins.object)
   Hello and goodbye
 
 Methods defined here:
-
__init__()
Wow, I have no function!
- -
-Data descriptors defined here:
-
__dict__
-
%s
-
-
__weakref__
-
%s
-
-

- - - -\x20\x20\x20\x20 - -
 
-class B(builtins.object)
    Data descriptors defined here:
-
__dict__
-
%s
-
-
__weakref__
-
%s
-
-
-Data and other attributes defined here:
-
NO_MEANING = 'eggs'
- -
__annotations__ = {'NO_MEANING': <class 'str'>}
- -

- - - -\x20\x20\x20\x20 - -
 
-class C(builtins.object)
    Methods defined here:
-
get_answer(self)
Return say_no()
- -
is_it_true(self)
Return self.get_answer()
- -
say_no(self)
- -
-Data descriptors defined here:
-
__dict__
-
dictionary for instance variables (if defined)
-
-
__weakref__
-
list of weak references to the object (if defined)
-
-

- - - -\x20\x20\x20\x20 - -
 
-Functions
       
doc_func()
This function solves all of the world's problems:
-hunger
-lack of Python
-war
-
nodoc_func()
-

- - - -\x20\x20\x20\x20 - -
 
-Data
       __xyz__ = 'X, Y and Z'

- - - -\x20\x20\x20\x20 - -
 
-Author
       Benjamin Peterson

- - - -\x20\x20\x20\x20 - -
 
-Credits
       Nobody
-""".strip() # ' <- emacs turd +html2text_of_expected = """ +test.pydoc_mod (version 1.2.3.4) +This is a test module for test_pydoc + +Classes + builtins.object + A + B + C + +class A(builtins.object) + Hello and goodbye + + Methods defined here: + __init__() + Wow, I have no function! + + Data descriptors defined here: + __dict__ + dictionary for instance variables (if defined) + __weakref__ + list of weak references to the object (if defined) + +class B(builtins.object) + Data descriptors defined here: + __dict__ + dictionary for instance variables (if defined) + __weakref__ + list of weak references to the object (if defined) + Data and other attributes defined here: + NO_MEANING = 'eggs' + __annotations__ = {'NO_MEANING': } + + +class C(builtins.object) + Methods defined here: + get_answer(self) + Return say_no() + is_it_true(self) + Return self.get_answer() + say_no(self) + Data descriptors defined here: + __dict__ + dictionary for instance variables (if defined) + __weakref__ + list of weak references to the object (if defined) + +Functions + doc_func() + This function solves all of the world's problems: + hunger + lack of Python + war + nodoc_func() + +Data + __xyz__ = 'X, Y and Z' + +Author + Benjamin Peterson + +Credits + Nobody +""" expected_html_data_docstrings = tuple(s.replace(' ', ' ') for s in expected_data_docstrings) @@ -394,6 +336,16 @@ def get_html_title(text): return title +def html2text(html): + """A quick and dirty implementation of html2text. + + Tailored for pydoc tests only. + """ + return pydoc.replace( + re.sub("<.*?>", "", html), + " ", " ", ">", ">", "<", "<") + + class PydocBaseTest(unittest.TestCase): def _restricted_walk_packages(self, walk_packages, path=None): @@ -434,12 +386,16 @@ class PydocDocTest(unittest.TestCase): @requires_docstrings def test_html_doc(self): result, doc_loc = get_pydoc_html(pydoc_mod) + text_result = html2text(result) + expected_lines = [line.strip() for line in html2text_of_expected if line] + for line in expected_lines: + self.assertIn(line, text_result) mod_file = inspect.getabsfile(pydoc_mod) mod_url = urllib.parse.quote(mod_file) - expected_html = expected_html_pattern % ( - (mod_url, mod_file, doc_loc) + - expected_html_data_docstrings) - self.assertEqual(result, expected_html) + self.assertIn(mod_url, result) + self.assertIn(mod_file, result) + self.assertIn(doc_loc, result) + @unittest.skipIf(sys.flags.optimize >= 2, "Docstrings are omitted with -O2 and above") @@ -845,47 +801,39 @@ class B(A) ''' % __name__) doc = pydoc.render_doc(B, renderer=pydoc.HTMLDoc()) - self.assertEqual(doc, '''\ -Python Library Documentation: class B in module %s + expected_text = """ +Python Library Documentation -

- - - -\x20\x20\x20\x20 - -
 
-class B(A)
    
Method resolution order:
-
B
-
A
-
builtins.object
-
-
-Methods defined here:
-
b_size = a_size(self)
- -
itemconfig = itemconfigure(self, tagOrId, cnf=None, **kw)
- -
itemconfigure(self, tagOrId, cnf=None, **kw)
Configure resources of an item TAGORID.
- -
-Methods inherited from A:
-
a_size(self)
Return size
- -
lift = tkraise(self, aboveThis=None)
- -
tkraise(self, aboveThis=None)
Raise this widget in the stacking order.
- -
-Data descriptors inherited from A:
-
__dict__
-
dictionary for instance variables (if defined)
-
-
__weakref__
-
list of weak references to the object (if defined)
-
-
\ -''' % __name__) +class B in module test.test_pydoc +class B(A) + Method resolution order: + B + A + builtins.object + + Methods defined here: + b_size = a_size(self) + itemconfig = itemconfigure(self, tagOrId, cnf=None, **kw) + itemconfigure(self, tagOrId, cnf=None, **kw) + Configure resources of an item TAGORID. + + Methods inherited from A: + a_size(self) + Return size + lift = tkraise(self, aboveThis=None) + tkraise(self, aboveThis=None) + Raise this widget in the stacking order. + + Data descriptors inherited from A: + __dict__ + dictionary for instance variables (if defined) + __weakref__ + list of weak references to the object (if defined) +""" + as_text = html2text(doc) + expected_lines = [line.strip() for line in expected_text.split("\n") if line] + for expected_line in expected_lines: + self.assertIn(expected_line, as_text) class PydocImportTest(PydocBaseTest): diff --git a/Lib/xmlrpc/server.py b/Lib/xmlrpc/server.py index 69a260f5b12c0..e22e480a829ff 100644 --- a/Lib/xmlrpc/server.py +++ b/Lib/xmlrpc/server.py @@ -440,7 +440,7 @@ class SimpleXMLRPCRequestHandler(BaseHTTPRequestHandler): # Class attribute listing the accessible path components; # paths not on this list will result in a 404 error. - rpc_paths = ('/', '/RPC2') + rpc_paths = ('/', '/RPC2', '/pydoc.css') #if not None, encode responses larger than this, if possible encode_threshold = 1400 #a common MTU @@ -801,7 +801,7 @@ def docserver(self, server_name, package_documentation, methods): server_name = self.escape(server_name) head = '%s' % server_name - result = self.heading(head, '#ffffff', '#7799ee') + result = self.heading(head) doc = self.markup(package_documentation, self.preformat, fdict) doc = doc and '%s' % doc @@ -812,10 +812,25 @@ def docserver(self, server_name, package_documentation, methods): for key, value in method_items: contents.append(self.docroutine(value, key, funcs=fdict)) result = result + self.bigsection( - 'Methods', '#ffffff', '#eeaa77', ''.join(contents)) + 'Methods', 'functions', ''.join(contents)) return result + + def page(self, title, contents): + """Format an HTML page.""" + css_path = "/pydoc.css" + css_link = ( + '' % + css_path) + return '''\ + + + + +Python: %s +%s%s''' % (title, css_link, contents) + class XMLRPCDocGenerator: """Generates documentation for an XML-RPC server. @@ -907,6 +922,12 @@ class DocXMLRPCRequestHandler(SimpleXMLRPCRequestHandler): for documentation. """ + def _get_css(self, url): + path_here = os.path.dirname(os.path.realpath(__file__)) + css_path = os.path.join(path_here, "..", "pydoc_data", "_pydoc.css") + with open(css_path, mode="rb") as fp: + return fp.read() + def do_GET(self): """Handles the HTTP GET request. @@ -918,9 +939,15 @@ def do_GET(self): self.report_404() return - response = self.server.generate_html_documentation().encode('utf-8') + if self.path.endswith('.css'): + content_type = 'text/css' + response = self._get_css(self.path) + else: + content_type = 'text/html' + response = self.server.generate_html_documentation().encode('utf-8') + self.send_response(200) - self.send_header("Content-type", "text/html") + self.send_header('Content-Type', '%s; charset=UTF-8' % content_type) self.send_header("Content-length", str(len(response))) self.end_headers() self.wfile.write(response) diff --git a/Misc/NEWS.d/next/Library/2021-10-08-04-11-55.bpo-10716.QSRVK2.rst b/Misc/NEWS.d/next/Library/2021-10-08-04-11-55.bpo-10716.QSRVK2.rst new file mode 100644 index 0000000000000..8ec944960211a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-08-04-11-55.bpo-10716.QSRVK2.rst @@ -0,0 +1,3 @@ +Migrated pydoc to HTML5 (without changing the look of it). Side effect is to +update xmlrpc's ``ServerHTMLDoc`` which now uses the CSS too. cgitb now +relies less on pydoc (as it can't use the CSS file). From webhook-mailer at python.org Sat Oct 9 04:25:10 2021 From: webhook-mailer at python.org (JulienPalard) Date: Sat, 09 Oct 2021 08:25:10 -0000 Subject: [Python-checkins] [doc]: update susp-ignored.csv after a98b273c. (GH-28827) Message-ID: https://github.com/python/cpython/commit/9fa930dd4859441d36e39ec6db1a6ebdc046d5b1 commit: 9fa930dd4859441d36e39ec6db1a6ebdc046d5b1 branch: main author: Julien Palard committer: JulienPalard date: 2021-10-09T10:25:00+02:00 summary: [doc]: update susp-ignored.csv after a98b273c. (GH-28827) files: M Doc/tools/susp-ignored.csv diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index b70e1ffced4e3d..0d9d109c0971e9 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -372,8 +372,8 @@ library/re,,`,"`" using/configure,84,:db2,=db1:db2:... library/typing,,`,# Type of ``val`` is narrowed to ``str`` library/typing,,`,"# Else, type of ``val`` is narrowed to ``float``." -library/typing,,`,# Type of ``val`` is narrowed to ``List[str]``. -library/typing,,`,# Type of ``val`` remains as ``List[object]``. +library/typing,,`,# Type of ``val`` is narrowed to ``list[str]``. +library/typing,,`,# Type of ``val`` remains as ``list[object]``. library/tkinter,,::,ttk::frame .frm -padding 10 library/tkinter,,::,"grid [ttk::label .frm.lbl -text ""Hello World!""] -column 0 -row 0" library/tkinter,,::,"grid [ttk::button .frm.btn -text ""Quit"" -command ""destroy .""] -column 1 -row 0" From webhook-mailer at python.org Sat Oct 9 09:17:35 2021 From: webhook-mailer at python.org (markshannon) Date: Sat, 09 Oct 2021 13:17:35 -0000 Subject: [Python-checkins] Bump MAGIC_NUMBER to reflect change in JUMP_ABSOLUTE semantics. (GH-28829) Message-ID: https://github.com/python/cpython/commit/5e173f5db17cbb2e3f2139a3c5ccb6b81ac59785 commit: 5e173f5db17cbb2e3f2139a3c5ccb6b81ac59785 branch: main author: Mark Shannon committer: markshannon date: 2021-10-09T14:17:22+01:00 summary: Bump MAGIC_NUMBER to reflect change in JUMP_ABSOLUTE semantics. (GH-28829) files: M Lib/importlib/_bootstrap_external.py diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index d624b73f38eb6..c9692b542a5cc 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -363,6 +363,7 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a1 3458 (imported objects now don't use LOAD_METHOD/CALL_METHOD) # Python 3.11a1 3459 (PEP 657: add end line numbers and column offsets for instructions) # Python 3.11a1 3460 (Add co_qualname field to PyCodeObject bpo-44530) +# Python 3.11a1 3461 (JUMP_ABSOLUTE must jump backwards) # # MAGIC must change whenever the bytecode emitted by the compiler may no @@ -372,7 +373,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3460).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3461).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' From webhook-mailer at python.org Sat Oct 9 10:50:21 2021 From: webhook-mailer at python.org (corona10) Date: Sat, 09 Oct 2021 14:50:21 -0000 Subject: [Python-checkins] bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) Message-ID: https://github.com/python/cpython/commit/34bbc87b2ddbaf245fbed6443c3e620f80c6a843 commit: 34bbc87b2ddbaf245fbed6443c3e620f80c6a843 branch: main author: Dong-hee Na committer: corona10 date: 2021-10-09T23:50:12+09:00 summary: bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) files: A Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst M Lib/test/test_csv.py M Modules/_csv.c diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 09e72a71f1db9..6e5dfc63d43ce 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -897,7 +897,7 @@ class mydialect(csv.Dialect): with self.assertRaises(csv.Error) as cm: mydialect() self.assertEqual(str(cm.exception), - '"quotechar" must be string, not int') + '"quotechar" must be string or None, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -934,6 +934,35 @@ class mydialect(csv.Dialect): self.assertEqual(str(cm.exception), '"delimiter" must be string, not int') + mydialect.delimiter = None + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not NoneType') + + def test_escapechar(self): + class mydialect(csv.Dialect): + delimiter = ";" + escapechar = '\\' + doublequote = False + skipinitialspace = True + lineterminator = '\r\n' + quoting = csv.QUOTE_NONE + d = mydialect() + self.assertEqual(d.escapechar, "\\") + + mydialect.escapechar = "**" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'): + mydialect() + + mydialect.escapechar = b"*" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not bytes'): + mydialect() + + mydialect.escapechar = 4 + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not int'): + mydialect() + def test_lineterminator(self): class mydialect(csv.Dialect): delimiter = ";" diff --git a/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst new file mode 100644 index 0000000000000..e75612116e942 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst @@ -0,0 +1,2 @@ +Improve error message of :class:`csv.Dialect` when initializing. +Patch by Vajrasky Kok and Dong-hee Na. diff --git a/Modules/_csv.c b/Modules/_csv.c index 3729d2be36252..cfdfbce6e6824 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -229,21 +229,21 @@ _set_int(const char *name, int *target, PyObject *src, int dflt) } static int -_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +_set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) { - if (src == NULL) + if (src == NULL) { *target = dflt; + } else { *target = '\0'; if (src != Py_None) { - Py_ssize_t len; if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, - "\"%s\" must be string, not %.200s", name, + "\"%s\" must be string or None, not %.200s", name, Py_TYPE(src)->tp_name); return -1; } - len = PyUnicode_GetLength(src); + Py_ssize_t len = PyUnicode_GetLength(src); if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -251,8 +251,38 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - if (len > 0) + else { *target = PyUnicode_READ_CHAR(src, 0); + } + } + } + return 0; +} + +static int +_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +{ + if (src == NULL) { + *target = dflt; + } + else { + *target = '\0'; + if (!PyUnicode_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + Py_TYPE(src)->tp_name); + return -1; + } + Py_ssize_t len = PyUnicode_GetLength(src); + if (len > 1) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be a 1-character string", + name); + return -1; + } + /* PyUnicode_READY() is called in PyUnicode_GetLength() */ + else { + *target = PyUnicode_READ_CHAR(src, 0); } } return 0; @@ -445,9 +475,9 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) goto err DIASET(_set_char, "delimiter", &self->delimiter, delimiter, ','); DIASET(_set_bool, "doublequote", &self->doublequote, doublequote, true); - DIASET(_set_char, "escapechar", &self->escapechar, escapechar, 0); + DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, 0); DIASET(_set_str, "lineterminator", &self->lineterminator, lineterminator, "\r\n"); - DIASET(_set_char, "quotechar", &self->quotechar, quotechar, '"'); + DIASET(_set_char_or_none, "quotechar", &self->quotechar, quotechar, '"'); DIASET(_set_int, "quoting", &self->quoting, quoting, QUOTE_MINIMAL); DIASET(_set_bool, "skipinitialspace", &self->skipinitialspace, skipinitialspace, false); DIASET(_set_bool, "strict", &self->strict, strict, false); From webhook-mailer at python.org Sat Oct 9 11:16:20 2021 From: webhook-mailer at python.org (corona10) Date: Sat, 09 Oct 2021 15:16:20 -0000 Subject: [Python-checkins] bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) Message-ID: https://github.com/python/cpython/commit/ec04db74e24a5f5da441bcabbe259157b4938b9b commit: ec04db74e24a5f5da441bcabbe259157b4938b9b branch: main author: Dong-hee Na committer: corona10 date: 2021-10-10T00:16:12+09:00 summary: bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) files: M Modules/_csv.c diff --git a/Modules/_csv.c b/Modules/_csv.c index cfdfbce6e6824..72f0791a4398b 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -244,6 +244,9 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -274,6 +277,9 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", From webhook-mailer at python.org Sat Oct 9 11:31:03 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 09 Oct 2021 15:31:03 -0000 Subject: [Python-checkins] bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) Message-ID: https://github.com/python/cpython/commit/6f3bc5eee6729197747d324c167da12902fb7c27 commit: 6f3bc5eee6729197747d324c167da12902fb7c27 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-09T08:30:54-07:00 summary: bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) (cherry picked from commit 34bbc87b2ddbaf245fbed6443c3e620f80c6a843) Co-authored-by: Dong-hee Na files: A Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst M Lib/test/test_csv.py M Modules/_csv.c diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 89fec4b22e07b..ce2eb72fd52fa 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -881,7 +881,7 @@ class mydialect(csv.Dialect): with self.assertRaises(csv.Error) as cm: mydialect() self.assertEqual(str(cm.exception), - '"quotechar" must be string, not int') + '"quotechar" must be string or None, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -918,6 +918,35 @@ class mydialect(csv.Dialect): self.assertEqual(str(cm.exception), '"delimiter" must be string, not int') + mydialect.delimiter = None + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not NoneType') + + def test_escapechar(self): + class mydialect(csv.Dialect): + delimiter = ";" + escapechar = '\\' + doublequote = False + skipinitialspace = True + lineterminator = '\r\n' + quoting = csv.QUOTE_NONE + d = mydialect() + self.assertEqual(d.escapechar, "\\") + + mydialect.escapechar = "**" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'): + mydialect() + + mydialect.escapechar = b"*" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not bytes'): + mydialect() + + mydialect.escapechar = 4 + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not int'): + mydialect() + def test_lineterminator(self): class mydialect(csv.Dialect): delimiter = ";" diff --git a/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst new file mode 100644 index 0000000000000..e75612116e942 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst @@ -0,0 +1,2 @@ +Improve error message of :class:`csv.Dialect` when initializing. +Patch by Vajrasky Kok and Dong-hee Na. diff --git a/Modules/_csv.c b/Modules/_csv.c index 7e06db46b496f..40a6361d2085f 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -231,21 +231,21 @@ _set_int(const char *name, int *target, PyObject *src, int dflt) } static int -_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +_set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) { - if (src == NULL) + if (src == NULL) { *target = dflt; + } else { *target = '\0'; if (src != Py_None) { - Py_ssize_t len; if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, - "\"%s\" must be string, not %.200s", name, + "\"%s\" must be string or None, not %.200s", name, Py_TYPE(src)->tp_name); return -1; } - len = PyUnicode_GetLength(src); + Py_ssize_t len = PyUnicode_GetLength(src); if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -253,8 +253,38 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - if (len > 0) + else { *target = PyUnicode_READ_CHAR(src, 0); + } + } + } + return 0; +} + +static int +_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +{ + if (src == NULL) { + *target = dflt; + } + else { + *target = '\0'; + if (!PyUnicode_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + Py_TYPE(src)->tp_name); + return -1; + } + Py_ssize_t len = PyUnicode_GetLength(src); + if (len > 1) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be a 1-character string", + name); + return -1; + } + /* PyUnicode_READY() is called in PyUnicode_GetLength() */ + else { + *target = PyUnicode_READ_CHAR(src, 0); } } return 0; @@ -423,9 +453,9 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) goto err DIASET(_set_char, "delimiter", &self->delimiter, delimiter, ','); DIASET(_set_bool, "doublequote", &self->doublequote, doublequote, true); - DIASET(_set_char, "escapechar", &self->escapechar, escapechar, 0); + DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, 0); DIASET(_set_str, "lineterminator", &self->lineterminator, lineterminator, "\r\n"); - DIASET(_set_char, "quotechar", &self->quotechar, quotechar, '"'); + DIASET(_set_char_or_none, "quotechar", &self->quotechar, quotechar, '"'); DIASET(_set_int, "quoting", &self->quoting, quoting, QUOTE_MINIMAL); DIASET(_set_bool, "skipinitialspace", &self->skipinitialspace, skipinitialspace, false); DIASET(_set_bool, "strict", &self->strict, strict, false); From webhook-mailer at python.org Sat Oct 9 11:35:37 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 09 Oct 2021 15:35:37 -0000 Subject: [Python-checkins] bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) Message-ID: https://github.com/python/cpython/commit/8772935765e7a4f04f7f561e37d0c0aee71d8030 commit: 8772935765e7a4f04f7f561e37d0c0aee71d8030 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-09T08:35:33-07:00 summary: bpo-20028: Improve error message of csv.Dialect when initializing (GH-28705) (cherry picked from commit 34bbc87b2ddbaf245fbed6443c3e620f80c6a843) Co-authored-by: Dong-hee Na files: A Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst M Lib/test/test_csv.py M Modules/_csv.c diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 09e72a71f1db9..6e5dfc63d43ce 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -897,7 +897,7 @@ class mydialect(csv.Dialect): with self.assertRaises(csv.Error) as cm: mydialect() self.assertEqual(str(cm.exception), - '"quotechar" must be string, not int') + '"quotechar" must be string or None, not int') def test_delimiter(self): class mydialect(csv.Dialect): @@ -934,6 +934,35 @@ class mydialect(csv.Dialect): self.assertEqual(str(cm.exception), '"delimiter" must be string, not int') + mydialect.delimiter = None + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"delimiter" must be string, not NoneType') + + def test_escapechar(self): + class mydialect(csv.Dialect): + delimiter = ";" + escapechar = '\\' + doublequote = False + skipinitialspace = True + lineterminator = '\r\n' + quoting = csv.QUOTE_NONE + d = mydialect() + self.assertEqual(d.escapechar, "\\") + + mydialect.escapechar = "**" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'): + mydialect() + + mydialect.escapechar = b"*" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not bytes'): + mydialect() + + mydialect.escapechar = 4 + with self.assertRaisesRegex(csv.Error, '"escapechar" must be string or None, not int'): + mydialect() + def test_lineterminator(self): class mydialect(csv.Dialect): delimiter = ";" diff --git a/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst new file mode 100644 index 0000000000000..e75612116e942 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-03-21-14-37.bpo-20028.zBA4RK.rst @@ -0,0 +1,2 @@ +Improve error message of :class:`csv.Dialect` when initializing. +Patch by Vajrasky Kok and Dong-hee Na. diff --git a/Modules/_csv.c b/Modules/_csv.c index 3729d2be36252..cfdfbce6e6824 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -229,21 +229,21 @@ _set_int(const char *name, int *target, PyObject *src, int dflt) } static int -_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +_set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) { - if (src == NULL) + if (src == NULL) { *target = dflt; + } else { *target = '\0'; if (src != Py_None) { - Py_ssize_t len; if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, - "\"%s\" must be string, not %.200s", name, + "\"%s\" must be string or None, not %.200s", name, Py_TYPE(src)->tp_name); return -1; } - len = PyUnicode_GetLength(src); + Py_ssize_t len = PyUnicode_GetLength(src); if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -251,8 +251,38 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - if (len > 0) + else { *target = PyUnicode_READ_CHAR(src, 0); + } + } + } + return 0; +} + +static int +_set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) +{ + if (src == NULL) { + *target = dflt; + } + else { + *target = '\0'; + if (!PyUnicode_Check(src)) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be string, not %.200s", name, + Py_TYPE(src)->tp_name); + return -1; + } + Py_ssize_t len = PyUnicode_GetLength(src); + if (len > 1) { + PyErr_Format(PyExc_TypeError, + "\"%s\" must be a 1-character string", + name); + return -1; + } + /* PyUnicode_READY() is called in PyUnicode_GetLength() */ + else { + *target = PyUnicode_READ_CHAR(src, 0); } } return 0; @@ -445,9 +475,9 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) goto err DIASET(_set_char, "delimiter", &self->delimiter, delimiter, ','); DIASET(_set_bool, "doublequote", &self->doublequote, doublequote, true); - DIASET(_set_char, "escapechar", &self->escapechar, escapechar, 0); + DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, 0); DIASET(_set_str, "lineterminator", &self->lineterminator, lineterminator, "\r\n"); - DIASET(_set_char, "quotechar", &self->quotechar, quotechar, '"'); + DIASET(_set_char_or_none, "quotechar", &self->quotechar, quotechar, '"'); DIASET(_set_int, "quoting", &self->quoting, quoting, QUOTE_MINIMAL); DIASET(_set_bool, "skipinitialspace", &self->skipinitialspace, skipinitialspace, false); DIASET(_set_bool, "strict", &self->strict, strict, false); From webhook-mailer at python.org Sat Oct 9 11:51:35 2021 From: webhook-mailer at python.org (pablogsal) Date: Sat, 09 Oct 2021 15:51:35 -0000 Subject: [Python-checkins] bpo-45256: Remove the usage of the C stack in Python to Python calls (GH-28488) Message-ID: https://github.com/python/cpython/commit/b4903afd4debbbd71dc49a2c8fefa74a3b6c6832 commit: b4903afd4debbbd71dc49a2c8fefa74a3b6c6832 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-09T16:51:30+01:00 summary: bpo-45256: Remove the usage of the C stack in Python to Python calls (GH-28488) Ths commit inlines calls to Python functions in the eval loop and steals all the arguments in the call from the caller for performance. files: M Include/internal/pycore_frame.h M Include/internal/pycore_tuple.h M Lib/test/gdb_sample.py M Lib/test/test_gdb.py M Objects/tupleobject.c M Python/ceval.c M Tools/gdb/libpython.py diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 489e115a8645c..7e63f584eb3b0 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -31,6 +31,7 @@ typedef struct _interpreter_frame { int f_lasti; /* Last instruction if called */ int stacktop; /* Offset of TOS from localsplus */ PyFrameState f_state; /* What state the frame is in */ + int depth; /* Depth of the frame in a ceval loop */ PyObject *localsplus[1]; } InterpreterFrame; @@ -85,6 +86,7 @@ _PyFrame_InitializeSpecials( frame->generator = NULL; frame->f_lasti = -1; frame->f_state = FRAME_CREATED; + frame->depth = 0; } /* Gets the pointer to the locals array diff --git a/Include/internal/pycore_tuple.h b/Include/internal/pycore_tuple.h index d1d0d2a92e49f..79c827fe8800a 100644 --- a/Include/internal/pycore_tuple.h +++ b/Include/internal/pycore_tuple.h @@ -13,6 +13,7 @@ extern "C" { #define _PyTuple_ITEMS(op) (_PyTuple_CAST(op)->ob_item) extern PyObject *_PyTuple_FromArray(PyObject *const *, Py_ssize_t); +extern PyObject *_PyTuple_FromArraySteal(PyObject *const *, Py_ssize_t); #ifdef __cplusplus } diff --git a/Lib/test/gdb_sample.py b/Lib/test/gdb_sample.py index cab13fb4da594..4188f50136fb9 100644 --- a/Lib/test/gdb_sample.py +++ b/Lib/test/gdb_sample.py @@ -1,7 +1,7 @@ # Sample script for use by test_gdb.py def foo(a, b, c): - bar(a, b, c) + bar(a=a, b=b, c=c) def bar(a, b, c): baz(a, b, c) diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index 22a8cf3c25fc0..fb0f1295574b9 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -734,8 +734,14 @@ def test_pyup_command(self): cmds_after_breakpoint=['py-up', 'py-up']) self.assertMultilineMatches(bt, r'''^.* +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) + id\(42\) #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) baz\(a, b, c\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\) + bar\(a=a, b=b, c=c\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in \(\) + foo\(1, 2, 3\) $''') @unittest.skipUnless(HAS_PYUP_PYDOWN, "test requires py-up/py-down commands") @@ -763,10 +769,18 @@ def test_up_then_down(self): cmds_after_breakpoint=['py-up', 'py-up', 'py-down']) self.assertMultilineMatches(bt, r'''^.* +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) + id\(42\) #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) baz\(a, b, c\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\) + bar\(a=a, b=b, c=c\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in \(\) + foo\(1, 2, 3\) #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) id\(42\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) + baz\(a, b, c\) $''') class PyBtTests(DebuggerTests): @@ -785,7 +799,7 @@ def test_bt(self): File ".*gdb_sample.py", line 7, in bar baz\(a, b, c\) File ".*gdb_sample.py", line 4, in foo - bar\(a, b, c\) + bar\(a=a, b=b, c=c\) File ".*gdb_sample.py", line 12, in foo\(1, 2, 3\) ''') @@ -801,7 +815,7 @@ def test_bt_full(self): #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) baz\(a, b, c\) #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\) - bar\(a, b, c\) + bar\(a=a, b=b, c=c\) #[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in \(\) foo\(1, 2, 3\) ''') @@ -1008,7 +1022,13 @@ def test_locals_after_up(self): bt = self.get_stack_trace(script=self.get_sample_script(), cmds_after_breakpoint=['py-up', 'py-up', 'py-locals']) self.assertMultilineMatches(bt, - r".*\na = 1\nb = 2\nc = 3\n.*") + r'''^.* +Locals for foo +a = 1 +b = 2 +c = 3 +Locals for +.*$''') def setUpModule(): diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c index e64b93ba63e41..018e738af06e3 100644 --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -484,6 +484,26 @@ _PyTuple_FromArray(PyObject *const *src, Py_ssize_t n) return (PyObject *)tuple; } +PyObject * +_PyTuple_FromArraySteal(PyObject *const *src, Py_ssize_t n) +{ + if (n == 0) { + return tuple_get_empty(); + } + + PyTupleObject *tuple = tuple_alloc(n); + if (tuple == NULL) { + return NULL; + } + PyObject **dst = tuple->ob_item; + for (Py_ssize_t i = 0; i < n; i++) { + PyObject *item = src[i]; + dst[i] = item; + } + _PyObject_GC_TRACK(tuple); + return (PyObject *)tuple; +} + static PyObject * tupleslice(PyTupleObject *a, Py_ssize_t ilow, Py_ssize_t ihigh) diff --git a/Python/ceval.c b/Python/ceval.c index e39ec67614bf5..6620c00d0f981 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -98,6 +98,12 @@ static int check_args_iterable(PyThreadState *, PyObject *func, PyObject *vararg static void format_kwargs_error(PyThreadState *, PyObject *func, PyObject *kwargs); static void format_awaitable_error(PyThreadState *, PyTypeObject *, int, int); static int get_exception_handler(PyCodeObject *, int, int*, int*, int*); +static InterpreterFrame * +_PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, + PyObject *locals, PyObject* const* args, + size_t argcount, PyObject *kwnames, int steal_args); +static int +_PyEvalFrameClearAndPop(PyThreadState *tstate, InterpreterFrame * frame); #define NAME_ERROR_MSG \ "name '%.200s' is not defined" @@ -1516,6 +1522,12 @@ trace_function_entry(PyThreadState *tstate, InterpreterFrame *frame) return 0; } +static PyObject * +make_coro(PyThreadState *tstate, PyFrameConstructor *con, + PyObject *locals, + PyObject* const* args, size_t argcount, + PyObject *kwnames); + static int skip_backwards_over_extended_args(PyCodeObject *code, int offset) { _Py_CODEUNIT *instrs = (_Py_CODEUNIT *)PyBytes_AS_STRING(code->co_code); @@ -1543,10 +1555,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *retval = NULL; /* Return value */ _Py_atomic_int * const eval_breaker = &tstate->interp->ceval.eval_breaker; - if (_Py_EnterRecursiveCall(tstate, "")) { - return NULL; - } - CFrame cframe; /* WARNING: Because the CFrame lives on the C stack, @@ -1558,9 +1566,18 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr cframe.previous = prev_cframe; tstate->cframe = &cframe; + assert(frame->depth == 0); /* push frame */ tstate->frame = frame; +start_frame: + if (_Py_EnterRecursiveCall(tstate, "")) { + tstate->recursion_depth++; + goto exit_eval_frame; + } + + assert(frame == tstate->frame); + if (cframe.use_tracing) { if (trace_function_entry(tstate, frame)) { goto exit_eval_frame; @@ -1582,7 +1599,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - +resume_frame: + co = frame->f_code; PyObject *names = co->co_names; PyObject *consts = co->co_consts; _Py_CODEUNIT *first_instr = co->co_firstinstr; @@ -1594,12 +1612,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr multiple values. When the PREDICT() macros are enabled, some opcode pairs follow in - direct succession without updating frame->f_lasti. A successful - prediction effectively links the two codes together as if they - were a single new opcode; accordingly,frame->f_lasti will point to - the first code in the pair (for instance, GET_ITER followed by - FOR_ITER is effectively a single opcode and frame->f_lasti will point - to the beginning of the combined pair.) + direct succession. A successful prediction effectively links the two + codes together as if they were a single new opcode, but the value + of frame->f_lasti is correctly updated so potential inlined calls + or lookups of frame->f_lasti are aways correct when the macros are used. */ assert(frame->f_lasti >= -1); _Py_CODEUNIT *next_instr = first_instr + frame->f_lasti + 1; @@ -1625,6 +1641,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr #endif if (throwflag) { /* support for generator.throw() */ + throwflag = 0; goto error; } @@ -4591,10 +4608,44 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(CALL_FUNCTION) { PREDICTED(CALL_FUNCTION); - PyObject **sp, *res; - sp = stack_pointer; - res = call_function(tstate, &sp, oparg, NULL, cframe.use_tracing); - stack_pointer = sp; + PyObject *res; + + // Check if the call can be inlined or not + PyObject *function = PEEK(oparg + 1); + if (Py_TYPE(function) == &PyFunction_Type) { + PyCodeObject *code = (PyCodeObject*)PyFunction_GET_CODE(function); + PyObject *locals = code->co_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); + if ((code->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) == 0) { + InterpreterFrame *new_frame = _PyEvalFramePushAndInit( + tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, stack_pointer-oparg, oparg, NULL, 1); + if (new_frame == NULL) { + // When we exit here, we own all variables in the stack (the frame creation has not stolen + // any variable) so we need to clean the whole stack (done in the "error" label). + goto error; + } + STACK_SHRINK(oparg + 1); + assert(tstate->interp->eval_frame != NULL); + // The frame has stolen all the arguments from the stack, so there is no need to clean them up.``` + Py_DECREF(function); + _PyFrame_SetStackPointer(frame, stack_pointer); + new_frame->depth = frame->depth + 1; + tstate->frame = frame = new_frame; + goto start_frame; + } + else { + /* Callable is a generator or coroutine function: create coroutine or generator. */ + res = make_coro(tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, stack_pointer-oparg, oparg, NULL); + STACK_SHRINK(oparg + 1); + for (int i = 0; i < oparg + 1; i++) { + Py_DECREF(stack_pointer[i]); + } + } + } + else { + PyObject **sp = stack_pointer; + res = call_function(tstate, &sp, oparg, NULL, cframe.use_tracing); + stack_pointer = sp; + } PUSH(res); if (res == NULL) { goto error; @@ -5018,14 +5069,28 @@ MISS_WITH_OPARG_COUNTER(BINARY_ADD) /* pop frame */ exit_eval_frame: - /* Restore previous cframe */ - tstate->cframe = cframe.previous; - tstate->cframe->use_tracing = cframe.use_tracing; - if (PyDTrace_FUNCTION_RETURN_ENABLED()) dtrace_function_return(frame); _Py_LeaveRecursiveCall(tstate); + + if (frame->depth) { + _PyFrame_StackPush(frame->previous, retval); + if (_PyEvalFrameClearAndPop(tstate, frame)) { + retval = NULL; + } + frame = tstate->frame; + if (retval == NULL) { + assert(_PyErr_Occurred(tstate)); + throwflag = 1; + } + retval = NULL; + goto resume_frame; + } tstate->frame = frame->previous; + + /* Restore previous cframe */ + tstate->cframe = cframe.previous; + tstate->cframe->use_tracing = cframe.use_tracing; return _Py_CheckFunctionResult(tstate, NULL, retval, __func__); } @@ -5336,7 +5401,7 @@ get_exception_handler(PyCodeObject *code, int index, int *level, int *handler, i static int initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, PyObject **localsplus, PyObject *const *args, - Py_ssize_t argcount, PyObject *kwnames) + Py_ssize_t argcount, PyObject *kwnames, int steal_args) { PyCodeObject *co = (PyCodeObject*)con->fc_code; const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount; @@ -5346,8 +5411,9 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, Py_ssize_t i; if (co->co_flags & CO_VARKEYWORDS) { kwdict = PyDict_New(); - if (kwdict == NULL) + if (kwdict == NULL) { goto fail; + } i = total_args; if (co->co_flags & CO_VARARGS) { i++; @@ -5369,14 +5435,21 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, } for (j = 0; j < n; j++) { PyObject *x = args[j]; - Py_INCREF(x); + if (!steal_args) { + Py_INCREF(x); + } assert(localsplus[j] == NULL); localsplus[j] = x; } /* Pack other positional arguments into the *args argument */ if (co->co_flags & CO_VARARGS) { - PyObject *u = _PyTuple_FromArray(args + n, argcount - n); + PyObject *u = NULL; + if (steal_args) { + u = _PyTuple_FromArraySteal(args + n, argcount - n); + } else { + u = _PyTuple_FromArray(args + n, argcount - n); + } if (u == NULL) { goto fail; } @@ -5442,6 +5515,9 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (PyDict_SetItem(kwdict, keyword, value) == -1) { goto fail; } + if (steal_args) { + Py_DECREF(value); + } continue; kw_found: @@ -5451,7 +5527,9 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, con->fc_qualname, keyword); goto fail; } - Py_INCREF(value); + if (!steal_args) { + Py_INCREF(value); + } localsplus[j] = value; } } @@ -5555,7 +5633,7 @@ make_coro_frame(PyThreadState *tstate, } _PyFrame_InitializeSpecials(frame, con, locals, code->co_nlocalsplus); assert(frame->frame_obj == NULL); - if (initialize_locals(tstate, con, frame->localsplus, args, argcount, kwnames)) { + if (initialize_locals(tstate, con, frame->localsplus, args, argcount, kwnames, 0)) { _PyFrame_Clear(frame, 1); return NULL; } @@ -5581,17 +5659,30 @@ make_coro(PyThreadState *tstate, PyFrameConstructor *con, return gen; } +// If *steal_args* is set, the function will steal the references to all the arguments. +// In case of error, the function returns null and if *steal_args* is set, the caller +// will still own all the arguments. static InterpreterFrame * _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals, PyObject* const* args, - size_t argcount, PyObject *kwnames) + size_t argcount, PyObject *kwnames, int steal_args) { InterpreterFrame * frame = _PyThreadState_PushFrame(tstate, con, locals); if (frame == NULL) { return NULL; } PyObject **localsarray = _PyFrame_GetLocalsArray(frame); - if (initialize_locals(tstate, con, localsarray, args, argcount, kwnames)) { + if (initialize_locals(tstate, con, localsarray, args, argcount, kwnames, steal_args)) { + if (steal_args) { + // If we failed to initialize locals, make sure the caller still own all the + // arguments. Notice that we only need to increase the reference count of the + // *valid* arguments (i.e. the ones that fit into the frame). + PyCodeObject *co = (PyCodeObject*)con->fc_code; + const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount; + for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) { + Py_XINCREF(frame->localsplus[i]); + } + } _PyFrame_Clear(frame, 0); return NULL; } @@ -5606,9 +5697,9 @@ _PyEvalFrameClearAndPop(PyThreadState *tstate, InterpreterFrame * frame) ++tstate->recursion_depth; assert(frame->frame_obj == NULL || frame->frame_obj->f_own_locals_memory == 0); if (_PyFrame_Clear(frame, 0)) { + --tstate->recursion_depth; return -1; } - assert(frame->frame_obj == NULL); --tstate->recursion_depth; tstate->frame = frame->previous; _PyThreadState_PopFrame(tstate, frame); @@ -5628,7 +5719,7 @@ _PyEval_Vector(PyThreadState *tstate, PyFrameConstructor *con, return make_coro(tstate, con, locals, args, argcount, kwnames); } InterpreterFrame *frame = _PyEvalFramePushAndInit( - tstate, con, locals, args, argcount, kwnames); + tstate, con, locals, args, argcount, kwnames, 0); if (frame == NULL) { return NULL; } diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index 62eb1976b715f..a118d326dc00e 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -45,6 +45,7 @@ # compatible (2.6+ and 3.0+). See #19308. from __future__ import print_function + import gdb import os import locale @@ -991,6 +992,11 @@ def _f_nlocalsplus(self): def _f_lasti(self): return self._f_special("f_lasti", int_from_int) + def depth(self): + return self._f_special("depth", int_from_int) + + def previous(self): + return self._f_special("previous", PyFramePtr) def iter_globals(self): ''' @@ -1797,16 +1803,20 @@ def get_selected_bytecode_frame(cls): def print_summary(self): if self.is_evalframe(): - pyop = self.get_pyop() - if pyop: - line = pyop.get_truncated_repr(MAX_OUTPUT_LEN) - write_unicode(sys.stdout, '#%i %s\n' % (self.get_index(), line)) - if not pyop.is_optimized_out(): - line = pyop.current_line() - if line is not None: - sys.stdout.write(' %s\n' % line.strip()) - else: - sys.stdout.write('#%i (unable to read python frame information)\n' % self.get_index()) + interp_frame = self.get_pyop() + while True: + if interp_frame: + line = interp_frame.get_truncated_repr(MAX_OUTPUT_LEN) + write_unicode(sys.stdout, '#%i %s\n' % (self.get_index(), line)) + if not interp_frame.is_optimized_out(): + line = interp_frame.current_line() + if line is not None: + sys.stdout.write(' %s\n' % line.strip()) + if interp_frame.depth() == 0: + break + else: + sys.stdout.write('#%i (unable to read python frame information)\n' % self.get_index()) + interp_frame = interp_frame.previous() else: info = self.is_other_python_frame() if info: @@ -1816,15 +1826,19 @@ def print_summary(self): def print_traceback(self): if self.is_evalframe(): - pyop = self.get_pyop() - if pyop: - pyop.print_traceback() - if not pyop.is_optimized_out(): - line = pyop.current_line() - if line is not None: - sys.stdout.write(' %s\n' % line.strip()) - else: - sys.stdout.write(' (unable to read python frame information)\n') + interp_frame = self.get_pyop() + while True: + if interp_frame: + interp_frame.print_traceback() + if not interp_frame.is_optimized_out(): + line = interp_frame.current_line() + if line is not None: + sys.stdout.write(' %s\n' % line.strip()) + if interp_frame.depth() == 0: + break + else: + sys.stdout.write(' (unable to read python frame information)\n') + interp_frame = interp_frame.previous() else: info = self.is_other_python_frame() if info: @@ -1914,11 +1928,15 @@ def invoke(self, args, from_tty): def move_in_stack(move_up): '''Move up or down the stack (for the py-up/py-down command)''' + # Important: + # The amount of frames that are printed out depends on how many frames are inlined + # in the same evaluation loop. As this command links directly the C stack with the + # Python stack, the results are sensitive to the number of inlined frames and this + # is likely to change between versions and optimizations. frame = Frame.get_selected_python_frame() if not frame: print('Unable to locate python frame') return - while frame: if move_up: iter_frame = frame.older() @@ -1940,9 +1958,10 @@ def move_in_stack(move_up): print('Unable to find an older python frame') else: print('Unable to find a newer python frame') + class PyUp(gdb.Command): - 'Select and print the python stack frame that called this one (if any)' + 'Select and print all python stack frame in the same eval loop starting from the one that called this one (if any)' def __init__(self): gdb.Command.__init__ (self, "py-up", @@ -1954,7 +1973,7 @@ def invoke(self, args, from_tty): move_in_stack(move_up=True) class PyDown(gdb.Command): - 'Select and print the python stack frame called by this one (if any)' + 'Select and print all python stack frame in the same eval loop starting from the one called this one (if any)' def __init__(self): gdb.Command.__init__ (self, "py-down", @@ -2067,13 +2086,20 @@ def invoke(self, args, from_tty): return pyop_frame = frame.get_pyop() - if not pyop_frame: - print(UNABLE_READ_INFO_PYTHON_FRAME) - return + while True: + if not pyop_frame: + print(UNABLE_READ_INFO_PYTHON_FRAME) + + sys.stdout.write('Locals for %s\n' % (pyop_frame.co_name.proxyval(set()))) + + for pyop_name, pyop_value in pyop_frame.iter_locals(): + print('%s = %s' + % (pyop_name.proxyval(set()), + pyop_value.get_truncated_repr(MAX_OUTPUT_LEN))) + + if pyop_frame.depth() == 0: + break - for pyop_name, pyop_value in pyop_frame.iter_locals(): - print('%s = %s' - % (pyop_name.proxyval(set()), - pyop_value.get_truncated_repr(MAX_OUTPUT_LEN))) + pyop_frame = pyop_frame.previous() PyLocals() From webhook-mailer at python.org Sat Oct 9 12:13:25 2021 From: webhook-mailer at python.org (corona10) Date: Sat, 09 Oct 2021 16:13:25 -0000 Subject: [Python-checkins] [3.10] bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) (GH-28834) Message-ID: https://github.com/python/cpython/commit/c80f0b7aa1d90332d0069d3e85ee112d0c9da7f0 commit: c80f0b7aa1d90332d0069d3e85ee112d0c9da7f0 branch: 3.10 author: Dong-hee Na committer: corona10 date: 2021-10-10T01:13:21+09:00 summary: [3.10] bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) (GH-28834) files: M Modules/_csv.c diff --git a/Modules/_csv.c b/Modules/_csv.c index cfdfbce6e6824..72f0791a4398b 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -244,6 +244,9 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -274,6 +277,9 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", From webhook-mailer at python.org Sat Oct 9 12:13:48 2021 From: webhook-mailer at python.org (corona10) Date: Sat, 09 Oct 2021 16:13:48 -0000 Subject: [Python-checkins] [3.9] bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) (GH-28835) Message-ID: https://github.com/python/cpython/commit/e4fcb6fd3dcc4db23867c2fbd538189b8f221225 commit: e4fcb6fd3dcc4db23867c2fbd538189b8f221225 branch: 3.9 author: Dong-hee Na committer: corona10 date: 2021-10-10T01:13:44+09:00 summary: [3.9] bpo-20028: Keep original exception when PyUnicode_GetLength return -1 (GH-28832) (GH-28835) files: M Modules/_csv.c diff --git a/Modules/_csv.c b/Modules/_csv.c index 40a6361d2085f..029f473ae8741 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -246,6 +246,9 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", @@ -276,6 +279,9 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } Py_ssize_t len = PyUnicode_GetLength(src); + if (len < 0) { + return -1; + } if (len > 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", From webhook-mailer at python.org Sat Oct 9 12:17:51 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 09 Oct 2021 16:17:51 -0000 Subject: [Python-checkins] bpo-27580: Add support of null characters in the csv module. (GH-28808) Message-ID: https://github.com/python/cpython/commit/b454e8e4df73bc73bc1a6f597431f171bfae8abd commit: b454e8e4df73bc73bc1a6f597431f171bfae8abd branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-09T19:17:43+03:00 summary: bpo-27580: Add support of null characters in the csv module. (GH-28808) files: A Misc/NEWS.d/next/Library/2021-10-07-21-11-48.bpo-27580.tGcBTH.rst M Lib/test/test_csv.py M Modules/_csv.c diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index 6e5dfc63d43ce..fb27ea396e04d 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -217,6 +217,17 @@ def test_write_escape(self): self._write_test(['C\\', '6', '7', 'X"'], 'C\\\\,6,7,"X"""', escapechar='\\', quoting=csv.QUOTE_MINIMAL) + def test_write_lineterminator(self): + for lineterminator in '\r\n', '\n', '\r', '!@#', '\0': + with self.subTest(lineterminator=lineterminator): + with StringIO() as sio: + writer = csv.writer(sio, lineterminator=lineterminator) + writer.writerow(['a', 'b']) + writer.writerow([1, 2]) + self.assertEqual(sio.getvalue(), + f'a,b{lineterminator}' + f'1,2{lineterminator}') + def test_write_iterable(self): self._write_test(iter(['a', 1, 'p,q']), 'a,1,"p,q"') self._write_test(iter(['a', 1, None]), 'a,1,') @@ -286,14 +297,10 @@ def test_read_oddinputs(self): self._read_test([''], [[]]) self.assertRaises(csv.Error, self._read_test, ['"ab"c'], None, strict = 1) - # cannot handle null bytes for the moment - self.assertRaises(csv.Error, self._read_test, - ['ab\0c'], None, strict = 1) self._read_test(['"ab"c'], [['abc']], doublequote = 0) self.assertRaises(csv.Error, self._read_test, - [b'ab\0c'], None) - + [b'abc'], None) def test_read_eol(self): self._read_test(['a,b'], [['a','b']]) @@ -313,6 +320,18 @@ def test_read_eof(self): self.assertRaises(csv.Error, self._read_test, ['^'], [], escapechar='^', strict=True) + def test_read_nul(self): + self._read_test(['\0'], [['\0']]) + self._read_test(['a,\0b,c'], [['a', '\0b', 'c']]) + self._read_test(['a,b\0,c'], [['a', 'b\0', 'c']]) + self._read_test(['a,b\\\0,c'], [['a', 'b\0', 'c']], escapechar='\\') + self._read_test(['a,"\0b",c'], [['a', '\0b', 'c']]) + + def test_read_delimiter(self): + self._read_test(['a,b,c'], [['a', 'b', 'c']]) + self._read_test(['a;b;c'], [['a', 'b', 'c']], delimiter=';') + self._read_test(['a\0b\0c'], [['a', 'b', 'c']], delimiter='\0') + def test_read_escape(self): self._read_test(['a,\\b,c'], [['a', 'b', 'c']], escapechar='\\') self._read_test(['a,b\\,c'], [['a', 'b,c']], escapechar='\\') @@ -320,6 +339,11 @@ def test_read_escape(self): self._read_test(['a,"b,\\c"'], [['a', 'b,c']], escapechar='\\') self._read_test(['a,"b,c\\""'], [['a', 'b,c"']], escapechar='\\') self._read_test(['a,"b,c"\\'], [['a', 'b,c\\']], escapechar='\\') + self._read_test(['a,^b,c'], [['a', 'b', 'c']], escapechar='^') + self._read_test(['a,\0b,c'], [['a', 'b', 'c']], escapechar='\0') + self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar=None) + self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar='') + self._read_test(['a,\\b,c'], [['a', '\\b', 'c']]) def test_read_quoting(self): self._read_test(['1,",3,",5'], [['1', ',3,', '5']]) @@ -334,6 +358,8 @@ def test_read_quoting(self): self.assertRaises(ValueError, self._read_test, ['abc,3'], [[]], quoting=csv.QUOTE_NONNUMERIC) + self._read_test(['1,@,3,@,5'], [['1', ',3,', '5']], quotechar='@') + self._read_test(['1,\0,3,\0,5'], [['1', ',3,', '5']], quotechar='\0') def test_read_bigfield(self): # This exercises the buffer realloc functionality and field size @@ -1074,6 +1100,12 @@ class TestSniffer(unittest.TestCase): a,b """) + sample14 = """\ +abc\0def +ghijkl\0mno +ghi\0jkl +""" + def test_issue43625(self): sniffer = csv.Sniffer() self.assertTrue(sniffer.has_header(self.sample12)) @@ -1142,6 +1174,8 @@ def test_delimiters(self): dialect = sniffer.sniff(self.sample9) self.assertEqual(dialect.delimiter, '+') self.assertEqual(dialect.quotechar, "'") + dialect = sniffer.sniff(self.sample14) + self.assertEqual(dialect.delimiter, '\0') def test_doublequote(self): sniffer = csv.Sniffer() diff --git a/Misc/NEWS.d/next/Library/2021-10-07-21-11-48.bpo-27580.tGcBTH.rst b/Misc/NEWS.d/next/Library/2021-10-07-21-11-48.bpo-27580.tGcBTH.rst new file mode 100644 index 0000000000000..15a8ff2dd6add --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-21-11-48.bpo-27580.tGcBTH.rst @@ -0,0 +1 @@ +Add support of null characters in :mod:`csv`. diff --git a/Modules/_csv.c b/Modules/_csv.c index 72f0791a4398b..469c1a15c340c 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -14,6 +14,9 @@ module instead. #include "structmember.h" // PyMemberDef #include +#define NOT_SET ((Py_UCS4)-1) +#define EOL ((Py_UCS4)-2) + typedef struct { PyObject *error_obj; /* CSV exception */ @@ -153,9 +156,9 @@ get_dialect_from_registry(PyObject *name_obj, _csvstate *module_state) } static PyObject * -get_nullchar_as_None(Py_UCS4 c) +get_char_or_None(Py_UCS4 c) { - if (c == '\0') { + if (c == NOT_SET) { Py_RETURN_NONE; } else @@ -172,19 +175,19 @@ Dialect_get_lineterminator(DialectObj *self, void *Py_UNUSED(ignored)) static PyObject * Dialect_get_delimiter(DialectObj *self, void *Py_UNUSED(ignored)) { - return get_nullchar_as_None(self->delimiter); + return get_char_or_None(self->delimiter); } static PyObject * Dialect_get_escapechar(DialectObj *self, void *Py_UNUSED(ignored)) { - return get_nullchar_as_None(self->escapechar); + return get_char_or_None(self->escapechar); } static PyObject * Dialect_get_quotechar(DialectObj *self, void *Py_UNUSED(ignored)) { - return get_nullchar_as_None(self->quotechar); + return get_char_or_None(self->quotechar); } static PyObject * @@ -235,7 +238,7 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt *target = dflt; } else { - *target = '\0'; + *target = NOT_SET; if (src != Py_None) { if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, @@ -254,7 +257,7 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - else { + else if (len > 0) { *target = PyUnicode_READ_CHAR(src, 0); } } @@ -269,7 +272,7 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) *target = dflt; } else { - *target = '\0'; + *target = NOT_SET; if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, "\"%s\" must be string, not %.200s", name, @@ -287,7 +290,7 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - else { + else if (len > 0) { *target = PyUnicode_READ_CHAR(src, 0); } } @@ -481,7 +484,7 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) goto err DIASET(_set_char, "delimiter", &self->delimiter, delimiter, ','); DIASET(_set_bool, "doublequote", &self->doublequote, doublequote, true); - DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, 0); + DIASET(_set_char_or_none, "escapechar", &self->escapechar, escapechar, NOT_SET); DIASET(_set_str, "lineterminator", &self->lineterminator, lineterminator, "\r\n"); DIASET(_set_char_or_none, "quotechar", &self->quotechar, quotechar, '"'); DIASET(_set_int, "quoting", &self->quoting, quoting, QUOTE_MINIMAL); @@ -491,19 +494,19 @@ dialect_new(PyTypeObject *type, PyObject *args, PyObject *kwargs) /* validate options */ if (dialect_check_quoting(self->quoting)) goto err; - if (self->delimiter == 0) { + if (self->delimiter == NOT_SET) { PyErr_SetString(PyExc_TypeError, "\"delimiter\" must be a 1-character string"); goto err; } if (quotechar == Py_None && quoting == NULL) self->quoting = QUOTE_NONE; - if (self->quoting != QUOTE_NONE && self->quotechar == 0) { + if (self->quoting != QUOTE_NONE && self->quotechar == NOT_SET) { PyErr_SetString(PyExc_TypeError, "quotechar must be set if quoting enabled"); goto err; } - if (self->lineterminator == 0) { + if (self->lineterminator == NULL) { PyErr_SetString(PyExc_TypeError, "lineterminator must be set"); goto err; } @@ -670,7 +673,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) switch (self->state) { case START_RECORD: /* start of record */ - if (c == '\0') + if (c == EOL) /* empty line - return [] */ break; else if (c == '\n' || c == '\r') { @@ -682,11 +685,11 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) /* fallthru */ case START_FIELD: /* expecting field */ - if (c == '\n' || c == '\r' || c == '\0') { + if (c == '\n' || c == '\r' || c == EOL) { /* save empty field - return [fields] */ if (parse_save_field(self) < 0) return -1; - self->state = (c == '\0' ? START_RECORD : EAT_CRNL); + self->state = (c == EOL ? START_RECORD : EAT_CRNL); } else if (c == dialect->quotechar && dialect->quoting != QUOTE_NONE) { @@ -722,7 +725,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) self->state = AFTER_ESCAPED_CRNL; break; } - if (c == '\0') + if (c == EOL) c = '\n'; if (parse_add_char(self, module_state, c) < 0) return -1; @@ -730,17 +733,17 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) break; case AFTER_ESCAPED_CRNL: - if (c == '\0') + if (c == EOL) break; /*fallthru*/ case IN_FIELD: /* in unquoted field */ - if (c == '\n' || c == '\r' || c == '\0') { + if (c == '\n' || c == '\r' || c == EOL) { /* end of line - return [fields] */ if (parse_save_field(self) < 0) return -1; - self->state = (c == '\0' ? START_RECORD : EAT_CRNL); + self->state = (c == EOL ? START_RECORD : EAT_CRNL); } else if (c == dialect->escapechar) { /* possible escaped character */ @@ -761,7 +764,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) case IN_QUOTED_FIELD: /* in quoted field */ - if (c == '\0') + if (c == EOL) ; else if (c == dialect->escapechar) { /* Possible escape character */ @@ -786,7 +789,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) break; case ESCAPE_IN_QUOTED_FIELD: - if (c == '\0') + if (c == EOL) c = '\n'; if (parse_add_char(self, module_state, c) < 0) return -1; @@ -808,11 +811,11 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) return -1; self->state = START_FIELD; } - else if (c == '\n' || c == '\r' || c == '\0') { + else if (c == '\n' || c == '\r' || c == EOL) { /* end of line - return [fields] */ if (parse_save_field(self) < 0) return -1; - self->state = (c == '\0' ? START_RECORD : EAT_CRNL); + self->state = (c == EOL ? START_RECORD : EAT_CRNL); } else if (!dialect->strict) { if (parse_add_char(self, module_state, c) < 0) @@ -831,7 +834,7 @@ parse_process_char(ReaderObj *self, _csvstate *module_state, Py_UCS4 c) case EAT_CRNL: if (c == '\n' || c == '\r') ; - else if (c == '\0') + else if (c == EOL) self->state = START_RECORD; else { PyErr_Format(module_state->error_obj, @@ -909,12 +912,6 @@ Reader_iternext(ReaderObj *self) linelen = PyUnicode_GET_LENGTH(lineobj); while (linelen--) { c = PyUnicode_READ(kind, data, pos); - if (c == '\0') { - Py_DECREF(lineobj); - PyErr_Format(module_state->error_obj, - "line contains NUL"); - goto err; - } if (parse_process_char(self, module_state, c) < 0) { Py_DECREF(lineobj); goto err; @@ -922,7 +919,7 @@ Reader_iternext(ReaderObj *self) pos++; } Py_DECREF(lineobj); - if (parse_process_char(self, module_state, 0) < 0) + if (parse_process_char(self, module_state, EOL) < 0) goto err; } while (self->state != START_RECORD); @@ -1127,7 +1124,7 @@ join_append_data(WriterObj *self, unsigned int field_kind, const void *field_dat *quoted = 1; } if (want_escape) { - if (!dialect->escapechar) { + if (dialect->escapechar == NOT_SET) { PyErr_Format(self->error_obj, "need to escape, but no escapechar set"); return -1; From webhook-mailer at python.org Sat Oct 9 12:52:10 2021 From: webhook-mailer at python.org (pablogsal) Date: Sat, 09 Oct 2021 16:52:10 -0000 Subject: [Python-checkins] bpo-45256: Small cleanups for the code that inlines Python-to-Python calls in ceval.c (GH-28836) Message-ID: https://github.com/python/cpython/commit/543acbce5a1e23633379a853f38dc55b12f6d931 commit: 543acbce5a1e23633379a853f38dc55b12f6d931 branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-09T17:52:05+01:00 summary: bpo-45256: Small cleanups for the code that inlines Python-to-Python calls in ceval.c (GH-28836) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 6620c00d0f981..df435bf3d7681 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4613,19 +4613,26 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr // Check if the call can be inlined or not PyObject *function = PEEK(oparg + 1); if (Py_TYPE(function) == &PyFunction_Type) { - PyCodeObject *code = (PyCodeObject*)PyFunction_GET_CODE(function); - PyObject *locals = code->co_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); - if ((code->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) == 0) { + int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(function))->co_flags; + PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); + int is_generator = code_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR); + if (!is_generator) { InterpreterFrame *new_frame = _PyEvalFramePushAndInit( - tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, stack_pointer-oparg, oparg, NULL, 1); + tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, + stack_pointer-oparg, + oparg, NULL, 1); if (new_frame == NULL) { - // When we exit here, we own all variables in the stack (the frame creation has not stolen - // any variable) so we need to clean the whole stack (done in the "error" label). + // When we exit here, we own all variables in the stack + // (the frame creation has not stolen any variable) so + // we need to clean the whole stack (done in the + // "error" label). goto error; } + STACK_SHRINK(oparg + 1); assert(tstate->interp->eval_frame != NULL); - // The frame has stolen all the arguments from the stack, so there is no need to clean them up.``` + // The frame has stolen all the arguments from the stack, + // so there is no need to clean them up. Py_DECREF(function); _PyFrame_SetStackPointer(frame, stack_pointer); new_frame->depth = frame->depth + 1; @@ -4633,8 +4640,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto start_frame; } else { - /* Callable is a generator or coroutine function: create coroutine or generator. */ - res = make_coro(tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, stack_pointer-oparg, oparg, NULL); + /* Callable is a generator or coroutine function: create + * coroutine or generator. */ + res = make_coro(tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), + locals, stack_pointer-oparg, oparg, NULL); STACK_SHRINK(oparg + 1); for (int i = 0; i < oparg + 1; i++) { Py_DECREF(stack_pointer[i]); @@ -4642,10 +4651,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } else { + /* Callable is not a Python function */ PyObject **sp = stack_pointer; res = call_function(tstate, &sp, oparg, NULL, cframe.use_tracing); stack_pointer = sp; } + PUSH(res); if (res == NULL) { goto error; @@ -5678,8 +5689,8 @@ _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, // arguments. Notice that we only need to increase the reference count of the // *valid* arguments (i.e. the ones that fit into the frame). PyCodeObject *co = (PyCodeObject*)con->fc_code; - const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount; - for (Py_ssize_t i = 0; i < Py_MIN(argcount, total_args); i++) { + const size_t total_args = co->co_argcount + co->co_kwonlyargcount; + for (size_t i = 0; i < Py_MIN(argcount, total_args); i++) { Py_XINCREF(frame->localsplus[i]); } } From webhook-mailer at python.org Sat Oct 9 14:47:04 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sat, 09 Oct 2021 18:47:04 -0000 Subject: [Python-checkins] Fix the "Finding all Adverbs" example (GH-21420) Message-ID: https://github.com/python/cpython/commit/dbd62e74dadda7868f1c0d497414c8f7e4c0b12b commit: dbd62e74dadda7868f1c0d497414c8f7e4c0b12b branch: main author: Rim Chatti committer: serhiy-storchaka date: 2021-10-09T21:46:56+03:00 summary: Fix the "Finding all Adverbs" example (GH-21420) Co-authored-by: Serhiy Storchaka files: M Doc/library/re.rst diff --git a/Doc/library/re.rst b/Doc/library/re.rst index ff7687cc936ec..b12ce4b9744f9 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1572,7 +1572,7 @@ find all of the adverbs in some text, they might use :func:`findall` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> re.findall(r"\w+ly", text) + >>> re.findall(r"\w+ly\b", text) ['carefully', 'quickly'] @@ -1586,7 +1586,7 @@ a writer wanted to find all of the adverbs *and their positions* in some text, they would use :func:`finditer` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> for m in re.finditer(r"\w+ly", text): + >>> for m in re.finditer(r"\w+ly\b", text): ... print('%02d-%02d: %s' % (m.start(), m.end(), m.group(0))) 07-16: carefully 40-47: quickly From webhook-mailer at python.org Sat Oct 9 15:18:00 2021 From: webhook-mailer at python.org (ericvsmith) Date: Sat, 09 Oct 2021 19:18:00 -0000 Subject: [Python-checkins] Fix dataclassses spelling (GH-28837) Message-ID: https://github.com/python/cpython/commit/5b4a7675bcfc6368aff955f4a6231579718f5dad commit: 5b4a7675bcfc6368aff955f4a6231579718f5dad branch: main author: Landon Yarrington <33426811+jly36963 at users.noreply.github.com> committer: ericvsmith date: 2021-10-09T15:17:52-04:00 summary: Fix dataclassses spelling (GH-28837) files: M Doc/whatsnew/3.10.rst M Lib/dataclasses.py M Lib/test/test_dataclasses.py diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index 266a80be3c96c..b1040d4888ed8 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -996,7 +996,7 @@ Added ``slots`` parameter in :func:`dataclasses.dataclass` decorator. Keyword-only fields ~~~~~~~~~~~~~~~~~~~ -dataclassses now supports fields that are keyword-only in the +dataclasses now supports fields that are keyword-only in the generated __init__ method. There are a number of ways of specifying keyword-only fields. diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py index 95ff39287bed6..1e98bf9b9bb97 100644 --- a/Lib/dataclasses.py +++ b/Lib/dataclasses.py @@ -1387,7 +1387,7 @@ def exec_body_callback(ns): ns['__annotations__'] = annotations # We use `types.new_class()` instead of simply `type()` to allow dynamic creation - # of generic dataclassses. + # of generic dataclasses. cls = types.new_class(cls_name, bases, {}, exec_body_callback) # Apply the normal decorator. diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index a1d9112135af3..bdcb4a2cfd1a0 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -1907,7 +1907,7 @@ class Parent(Generic[T]): # Check MRO resolution. self.assertEqual(Child.__mro__, (Child, Parent, Generic, object)) - def test_dataclassses_pickleable(self): + def test_dataclasses_pickleable(self): global P, Q, R @dataclass class P: From webhook-mailer at python.org Sat Oct 9 15:34:17 2021 From: webhook-mailer at python.org (gpshead) Date: Sat, 09 Oct 2021 19:34:17 -0000 Subject: [Python-checkins] bpo-45353: Remind sys.modules users to copy when iterating. (GH-28842) Message-ID: https://github.com/python/cpython/commit/3d1ca867ed0e3ae343166806f8ddd9739e568ab4 commit: 3d1ca867ed0e3ae343166806f8ddd9739e568ab4 branch: main author: Gregory P. Smith committer: gpshead date: 2021-10-09T12:34:13-07:00 summary: bpo-45353: Remind sys.modules users to copy when iterating. (GH-28842) This is true of all dictionaries in Python, but this one tends to catch people off guard as they don't realize when sys.modules might change out from underneath them as a hidden side effect of their code. Copying it first avoids the RuntimeError. An example when this happens in single threaded code are codecs being loaded which are an implicit time of use import that most need not think about. files: M Doc/library/sys.rst diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index ec12e02fb37d4..8b3c6fd762731 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1073,7 +1073,11 @@ always available. This is a dictionary that maps module names to modules which have already been loaded. This can be manipulated to force reloading of modules and other tricks. However, replacing the dictionary will not necessarily work as expected and - deleting essential items from the dictionary may cause Python to fail. + deleting essential items from the dictionary may cause Python to fail. If + you want to iterate over this global dictionary always use + ``sys.modules.copy()`` or ``tuple(sys.modules)`` to avoid exceptions as its + size may change during iteration as a side effect of code or activity in + other threads. .. data:: orig_argv From webhook-mailer at python.org Sat Oct 9 15:50:54 2021 From: webhook-mailer at python.org (ericvsmith) Date: Sat, 09 Oct 2021 19:50:54 -0000 Subject: [Python-checkins] Fix dataclassses spelling (GH-28837) (GH-28841) Message-ID: https://github.com/python/cpython/commit/e086bfee035eb23397f3cddba07b767b35d7ec08 commit: e086bfee035eb23397f3cddba07b767b35d7ec08 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ericvsmith date: 2021-10-09T15:50:45-04:00 summary: Fix dataclassses spelling (GH-28837) (GH-28841) (cherry picked from commit 5b4a7675bcfc6368aff955f4a6231579718f5dad) Co-authored-by: Landon Yarrington <33426811+jly36963 at users.noreply.github.com> Co-authored-by: Landon Yarrington <33426811+jly36963 at users.noreply.github.com> files: M Doc/whatsnew/3.10.rst M Lib/dataclasses.py M Lib/test/test_dataclasses.py diff --git a/Doc/whatsnew/3.10.rst b/Doc/whatsnew/3.10.rst index ddcefb408b7f8..5ac45c50c27c3 100644 --- a/Doc/whatsnew/3.10.rst +++ b/Doc/whatsnew/3.10.rst @@ -996,7 +996,7 @@ Added ``slots`` parameter in :func:`dataclasses.dataclass` decorator. Keyword-only fields ~~~~~~~~~~~~~~~~~~~ -dataclassses now supports fields that are keyword-only in the +dataclasses now supports fields that are keyword-only in the generated __init__ method. There are a number of ways of specifying keyword-only fields. diff --git a/Lib/dataclasses.py b/Lib/dataclasses.py index 79739976d3d29..aa84f1b9533cb 100644 --- a/Lib/dataclasses.py +++ b/Lib/dataclasses.py @@ -1387,7 +1387,7 @@ def exec_body_callback(ns): ns['__annotations__'] = annotations # We use `types.new_class()` instead of simply `type()` to allow dynamic creation - # of generic dataclassses. + # of generic dataclasses. cls = types.new_class(cls_name, bases, {}, exec_body_callback) # Apply the normal decorator. diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index a1d9112135af3..bdcb4a2cfd1a0 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -1907,7 +1907,7 @@ class Parent(Generic[T]): # Check MRO resolution. self.assertEqual(Child.__mro__, (Child, Parent, Generic, object)) - def test_dataclassses_pickleable(self): + def test_dataclasses_pickleable(self): global P, Q, R @dataclass class P: From webhook-mailer at python.org Sat Oct 9 15:54:30 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 09 Oct 2021 19:54:30 -0000 Subject: [Python-checkins] bpo-45353: Remind sys.modules users to copy when iterating. (GH-28842) Message-ID: https://github.com/python/cpython/commit/459a4db5eae1f5ef063b34c61cc099820aa9ed0a commit: 459a4db5eae1f5ef063b34c61cc099820aa9ed0a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-09T12:54:20-07:00 summary: bpo-45353: Remind sys.modules users to copy when iterating. (GH-28842) This is true of all dictionaries in Python, but this one tends to catch people off guard as they don't realize when sys.modules might change out from underneath them as a hidden side effect of their code. Copying it first avoids the RuntimeError. An example when this happens in single threaded code are codecs being loaded which are an implicit time of use import that most need not think about. (cherry picked from commit 3d1ca867ed0e3ae343166806f8ddd9739e568ab4) Co-authored-by: Gregory P. Smith files: M Doc/library/sys.rst diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index ec12e02fb37d4..8b3c6fd762731 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1073,7 +1073,11 @@ always available. This is a dictionary that maps module names to modules which have already been loaded. This can be manipulated to force reloading of modules and other tricks. However, replacing the dictionary will not necessarily work as expected and - deleting essential items from the dictionary may cause Python to fail. + deleting essential items from the dictionary may cause Python to fail. If + you want to iterate over this global dictionary always use + ``sys.modules.copy()`` or ``tuple(sys.modules)`` to avoid exceptions as its + size may change during iteration as a side effect of code or activity in + other threads. .. data:: orig_argv From webhook-mailer at python.org Sun Oct 10 03:14:47 2021 From: webhook-mailer at python.org (JulienPalard) Date: Sun, 10 Oct 2021 07:14:47 -0000 Subject: [Python-checkins] Fix EncodingWarning in test_tools. (GH-28846) Message-ID: https://github.com/python/cpython/commit/a1c3c9e8245a88cf37b084869b3559638116daf7 commit: a1c3c9e8245a88cf37b084869b3559638116daf7 branch: main author: Inada Naoki committer: JulienPalard date: 2021-10-10T09:14:40+02:00 summary: Fix EncodingWarning in test_tools. (GH-28846) files: M Lib/test/test_tools/test_fixcid.py M Lib/test/test_tools/test_gprof2html.py M Lib/test/test_tools/test_i18n.py M Lib/test/test_tools/test_lll.py M Lib/test/test_tools/test_pdeps.py M Lib/test/test_tools/test_pindent.py M Tools/scripts/gprof2html.py diff --git a/Lib/test/test_tools/test_fixcid.py b/Lib/test/test_tools/test_fixcid.py index 3df1368043719..a72f74b8b15b3 100644 --- a/Lib/test/test_tools/test_fixcid.py +++ b/Lib/test/test_tools/test_fixcid.py @@ -61,9 +61,10 @@ def test_directory(self): os.mkdir(os_helper.TESTFN) self.addCleanup(os_helper.rmtree, os_helper.TESTFN) c_filename = os.path.join(os_helper.TESTFN, "file.c") - with open(c_filename, "w") as file: + with open(c_filename, "w", encoding="utf-8") as file: file.write("int xx;\n") - with open(os.path.join(os_helper.TESTFN, "file.py"), "w") as file: + with open(os.path.join(os_helper.TESTFN, "file.py"), "w", + encoding="utf-8") as file: file.write("xx = 'unaltered'\n") script = os.path.join(scriptsdir, "fixcid.py") output = self.run_script(args=(os_helper.TESTFN,)) @@ -76,7 +77,7 @@ def test_directory(self): def run_script(self, input="", *, args=("-",), substfile="xx yy\n"): substfilename = os_helper.TESTFN + ".subst" - with open(substfilename, "w") as file: + with open(substfilename, "w", encoding="utf-8") as file: file.write(substfile) self.addCleanup(os_helper.unlink, substfilename) diff --git a/Lib/test/test_tools/test_gprof2html.py b/Lib/test/test_tools/test_gprof2html.py index 9489ed91f9c1c..7cceb8faf8e5f 100644 --- a/Lib/test/test_tools/test_gprof2html.py +++ b/Lib/test/test_tools/test_gprof2html.py @@ -25,7 +25,7 @@ def test_gprof(self): with mock.patch.object(self.gprof, 'webbrowser') as wmock, \ tempfile.TemporaryDirectory() as tmpdir: fn = os.path.join(tmpdir, 'abc') - open(fn, 'w').close() + open(fn, 'wb').close() sys.argv = ['gprof2html', fn] self.gprof.main() self.assertTrue(wmock.open.called) diff --git a/Lib/test/test_tools/test_i18n.py b/Lib/test/test_tools/test_i18n.py index 12f778dbf8405..7f18edaaa8ca6 100644 --- a/Lib/test/test_tools/test_i18n.py +++ b/Lib/test/test_tools/test_i18n.py @@ -57,10 +57,10 @@ def extract_docstrings_from_str(self, module_content): """ utility: return all msgids extracted from module_content """ filename = 'test_docstrings.py' with temp_cwd(None) as cwd: - with open(filename, 'w') as fp: + with open(filename, 'w', encoding='utf-8') as fp: fp.write(module_content) assert_python_ok(self.script, '-D', filename) - with open('messages.pot') as fp: + with open('messages.pot', encoding='utf-8') as fp: data = fp.read() return self.get_msgids(data) @@ -70,7 +70,7 @@ def test_header(self): """ with temp_cwd(None) as cwd: assert_python_ok(self.script) - with open('messages.pot') as fp: + with open('messages.pot', encoding='utf-8') as fp: data = fp.read() header = self.get_header(data) @@ -97,7 +97,7 @@ def test_POT_Creation_Date(self): from datetime import datetime with temp_cwd(None) as cwd: assert_python_ok(self.script) - with open('messages.pot') as fp: + with open('messages.pot', encoding='utf-8') as fp: data = fp.read() header = self.get_header(data) creationDate = header['POT-Creation-Date'] @@ -299,16 +299,19 @@ def test_files_list(self): text3 = 'Text to ignore' with temp_cwd(None), temp_dir(None) as sdir: os.mkdir(os.path.join(sdir, 'pypkg')) - with open(os.path.join(sdir, 'pypkg', 'pymod.py'), 'w') as sfile: + with open(os.path.join(sdir, 'pypkg', 'pymod.py'), 'w', + encoding='utf-8') as sfile: sfile.write(f'_({text1!r})') os.mkdir(os.path.join(sdir, 'pkg.py')) - with open(os.path.join(sdir, 'pkg.py', 'pymod2.py'), 'w') as sfile: + with open(os.path.join(sdir, 'pkg.py', 'pymod2.py'), 'w', + encoding='utf-8') as sfile: sfile.write(f'_({text2!r})') os.mkdir(os.path.join(sdir, 'CVS')) - with open(os.path.join(sdir, 'CVS', 'pymod3.py'), 'w') as sfile: + with open(os.path.join(sdir, 'CVS', 'pymod3.py'), 'w', + encoding='utf-8') as sfile: sfile.write(f'_({text3!r})') assert_python_ok(self.script, sdir) - with open('messages.pot') as fp: + with open('messages.pot', encoding='utf-8') as fp: data = fp.read() self.assertIn(f'msgid "{text1}"', data) self.assertIn(f'msgid "{text2}"', data) diff --git a/Lib/test/test_tools/test_lll.py b/Lib/test/test_tools/test_lll.py index ec0c97334fdeb..6eeb96ed9b67e 100644 --- a/Lib/test/test_tools/test_lll.py +++ b/Lib/test/test_tools/test_lll.py @@ -22,7 +22,7 @@ def test_lll_multiple_dirs(self): fn1 = os.path.join(dir1, 'foo1') fn2 = os.path.join(dir2, 'foo2') for fn, dir in (fn1, dir1), (fn2, dir2): - open(fn, 'w').close() + open(fn, 'wb').close() os.symlink(fn, os.path.join(dir, 'symlink')) with support.captured_stdout() as output: diff --git a/Lib/test/test_tools/test_pdeps.py b/Lib/test/test_tools/test_pdeps.py index 27cbfe215d039..a986d10e499d9 100644 --- a/Lib/test/test_tools/test_pdeps.py +++ b/Lib/test/test_tools/test_pdeps.py @@ -19,7 +19,7 @@ def test_process_errors(self): # Issue #14492: m_import.match(line) can be None. with tempfile.TemporaryDirectory() as tmpdir: fn = os.path.join(tmpdir, 'foo') - with open(fn, 'w') as stream: + with open(fn, 'w', encoding='utf-8') as stream: stream.write("#!/this/will/fail") self.pdeps.process(fn, {}) diff --git a/Lib/test/test_tools/test_pindent.py b/Lib/test/test_tools/test_pindent.py index e7a547ad7d612..01f13850eae73 100644 --- a/Lib/test/test_tools/test_pindent.py +++ b/Lib/test/test_tools/test_pindent.py @@ -37,9 +37,9 @@ def test_selftest(self): self.maxDiff = None with os_helper.temp_dir() as directory: data_path = os.path.join(directory, '_test.py') - with open(self.script) as f: + with open(self.script, encoding='utf-8') as f: closed = f.read() - with open(data_path, 'w') as f: + with open(data_path, 'w', encoding='utf-8') as f: f.write(closed) rc, out, err = assert_python_ok(self.script, '-d', data_path) @@ -47,9 +47,9 @@ def test_selftest(self): self.assertEqual(err, b'') backup = data_path + '~' self.assertTrue(os.path.exists(backup)) - with open(backup) as f: + with open(backup, encoding='utf-8') as f: self.assertEqual(f.read(), closed) - with open(data_path) as f: + with open(data_path, encoding='utf-8') as f: clean = f.read() compile(clean, '_test.py', 'exec') self.assertEqual(self.pindent(clean, '-c'), closed) @@ -58,20 +58,20 @@ def test_selftest(self): rc, out, err = assert_python_ok(self.script, '-c', data_path) self.assertEqual(out, b'') self.assertEqual(err, b'') - with open(backup) as f: + with open(backup, encoding='utf-8') as f: self.assertEqual(f.read(), clean) - with open(data_path) as f: + with open(data_path, encoding='utf-8') as f: self.assertEqual(f.read(), closed) broken = self.lstriplines(closed) - with open(data_path, 'w') as f: + with open(data_path, 'w', encoding='utf-8') as f: f.write(broken) rc, out, err = assert_python_ok(self.script, '-r', data_path) self.assertEqual(out, b'') self.assertEqual(err, b'') - with open(backup) as f: + with open(backup, encoding='utf-8') as f: self.assertEqual(f.read(), broken) - with open(data_path) as f: + with open(data_path, encoding='utf-8') as f: indented = f.read() compile(indented, '_test.py', 'exec') self.assertEqual(self.pindent(broken, '-r'), indented) diff --git a/Tools/scripts/gprof2html.py b/Tools/scripts/gprof2html.py index b14def4ef8482..bf0530ef3e433 100755 --- a/Tools/scripts/gprof2html.py +++ b/Tools/scripts/gprof2html.py @@ -24,7 +24,7 @@ """ def add_escapes(filename): - with open(filename) as fp: + with open(filename, encoding="utf-8") as fp: for line in fp: yield html.escape(line) @@ -79,7 +79,7 @@ def main(): filename = sys.argv[1] outputfilename = filename + ".html" input = add_escapes(filename) - with open(outputfilename, "w") as output: + with open(outputfilename, "w", encoding="utf-8") as output: gprof2html(input, output, filename) webbrowser.open("file:" + os.path.abspath(outputfilename)) From webhook-mailer at python.org Sun Oct 10 04:29:51 2021 From: webhook-mailer at python.org (methane) Date: Sun, 10 Oct 2021 08:29:51 -0000 Subject: [Python-checkins] bpo-29410: Change the default hash algorithm to SipHash13. (GH-28752) Message-ID: https://github.com/python/cpython/commit/ad970e8623523a8656e8c1ff4e1dff3423498a5a commit: ad970e8623523a8656e8c1ff4e1dff3423498a5a branch: main author: Inada Naoki committer: methane date: 2021-10-10T17:29:46+09:00 summary: bpo-29410: Change the default hash algorithm to SipHash13. (GH-28752) Co-authored-by: Erlend Egeberg Aasland Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Core and Builtins/2021-10-07-19-09-12.bpo-29410.bg5SYp.rst M Doc/using/configure.rst M Doc/whatsnew/3.11.rst M Include/pyhash.h M Lib/test/test_hash.py M Lib/test/test_imp.py M Lib/test/test_sys.py M Python/pyhash.c M configure M configure.ac M pyconfig.h.in diff --git a/Doc/using/configure.rst b/Doc/using/configure.rst index a59bee4ccf0f6..75f572c61877f 100644 --- a/Doc/using/configure.rst +++ b/Doc/using/configure.rst @@ -416,15 +416,19 @@ Libraries options Security Options ---------------- -.. cmdoption:: --with-hash-algorithm=[fnv|siphash24] +.. cmdoption:: --with-hash-algorithm=[fnv|siphash13|siphash24] Select hash algorithm for use in ``Python/pyhash.c``: - * ``siphash24`` (default). - * ``fnv``; + * ``siphash13`` (default); + * ``siphash24``; + * ``fnv``. .. versionadded:: 3.4 + .. versionadded:: 3.11 + ``siphash13`` is added and it is the new default. + .. cmdoption:: --with-builtin-hashlib-hashes=md5,sha1,sha256,sha512,sha3,blake2 Built-in hash modules: diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index ff376d231bafa..e3650a6da8b95 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -175,6 +175,11 @@ Other CPython Implementation Changes support :class:`typing.SupportsComplex` and :class:`typing.SupportsBytes` protocols. (Contributed by Mark Dickinson and Dong-hee Na in :issue:`24234`.) +* ``siphash13`` is added as a new internal hashing algorithms. It's has similar security + properties as ``siphash24`` but it is slightly faster for long inputs. ``str``, ``bytes``, + and some other types now use it as default algorithm for ``hash()``. :pep:`552` + hash-based pyc files now use ``siphash13``, too. + (Contributed by Inada Naoki in :issue:`29410`.) New Modules =========== diff --git a/Include/pyhash.h b/Include/pyhash.h index a314ea907b7fe..182d223fab1ca 100644 --- a/Include/pyhash.h +++ b/Include/pyhash.h @@ -114,11 +114,10 @@ PyAPI_FUNC(PyHash_FuncDef*) PyHash_GetFuncDef(void); /* hash algorithm selection * - * The values for Py_HASH_SIPHASH24 and Py_HASH_FNV are hard-coded in the + * The values for Py_HASH_* are hard-coded in the * configure script. * - * - FNV is available on all platforms and architectures. - * - SIPHASH24 only works on platforms that don't require aligned memory for integers. + * - FNV and SIPHASH* are available on all platforms and architectures. * - With EXTERNAL embedders can provide an alternative implementation with:: * * PyHash_FuncDef PyHash_Func = {...}; @@ -128,10 +127,11 @@ PyAPI_FUNC(PyHash_FuncDef*) PyHash_GetFuncDef(void); #define Py_HASH_EXTERNAL 0 #define Py_HASH_SIPHASH24 1 #define Py_HASH_FNV 2 +#define Py_HASH_SIPHASH13 3 #ifndef Py_HASH_ALGORITHM # ifndef HAVE_ALIGNED_REQUIRED -# define Py_HASH_ALGORITHM Py_HASH_SIPHASH24 +# define Py_HASH_ALGORITHM Py_HASH_SIPHASH13 # else # define Py_HASH_ALGORITHM Py_HASH_FNV # endif /* uint64_t && uint32_t && aligned */ diff --git a/Lib/test/test_hash.py b/Lib/test/test_hash.py index c68e0d8f81537..cf9db66a29ae1 100644 --- a/Lib/test/test_hash.py +++ b/Lib/test/test_hash.py @@ -42,8 +42,8 @@ def pysiphash(uint64): def skip_unless_internalhash(test): """Skip decorator for tests that depend on SipHash24 or FNV""" - ok = sys.hash_info.algorithm in {"fnv", "siphash24"} - msg = "Requires SipHash24 or FNV" + ok = sys.hash_info.algorithm in {"fnv", "siphash13", "siphash24"} + msg = "Requires SipHash13, SipHash24 or FNV" return test if ok else unittest.skip(msg)(test) @@ -206,6 +206,19 @@ class StringlikeHashRandomizationTests(HashRandomizationTests): # seed 42, 'abc' [-678966196, 573763426263223372, -820489388, -4282905804826039665], ], + 'siphash13': [ + # NOTE: PyUCS2 layout depends on endianness + # seed 0, 'abc' + [69611762, -4594863902769663758, 69611762, -4594863902769663758], + # seed 42, 'abc' + [-975800855, 3869580338025362921, -975800855, 3869580338025362921], + # seed 42, 'abcdefghijk' + [-595844228, 7764564197781545852, -595844228, 7764564197781545852], + # seed 0, '????' + [-1093288643, -2810468059467891395, -1041341092, 4925090034378237276], + # seed 42, '????' + [-585999602, -2845126246016066802, -817336969, -2219421378907968137], + ], 'siphash24': [ # NOTE: PyUCS2 layout depends on endianness # seed 0, 'abc' diff --git a/Lib/test/test_imp.py b/Lib/test/test_imp.py index 99312cc1625c4..1a21025fe6eaf 100644 --- a/Lib/test/test_imp.py +++ b/Lib/test/test_imp.py @@ -351,8 +351,8 @@ def test_issue_35321(self): self.assertEqual(_frozen_importlib.__spec__.origin, "frozen") def test_source_hash(self): - self.assertEqual(_imp.source_hash(42, b'hi'), b'\xc6\xe7Z\r\x03:}\xab') - self.assertEqual(_imp.source_hash(43, b'hi'), b'\x85\x9765\xf8\x9a\x8b9') + self.assertEqual(_imp.source_hash(42, b'hi'), b'\xfb\xd9G\x05\xaf$\x9b~') + self.assertEqual(_imp.source_hash(43, b'hi'), b'\xd0/\x87C\xccC\xff\xe2') def test_pyc_invalidation_mode_from_cmdline(self): cases = [ diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 6720cd622d7c5..93e39bc1836e0 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -508,7 +508,7 @@ def test_attributes(self): self.assertIsInstance(sys.hash_info.nan, int) self.assertIsInstance(sys.hash_info.imag, int) algo = sysconfig.get_config_var("Py_HASH_ALGORITHM") - if sys.hash_info.algorithm in {"fnv", "siphash24"}: + if sys.hash_info.algorithm in {"fnv", "siphash13", "siphash24"}: self.assertIn(sys.hash_info.hash_bits, {32, 64}) self.assertIn(sys.hash_info.seed_bits, {32, 64, 128}) @@ -516,8 +516,10 @@ def test_attributes(self): self.assertEqual(sys.hash_info.algorithm, "siphash24") elif algo == 2: self.assertEqual(sys.hash_info.algorithm, "fnv") + elif algo == 3: + self.assertEqual(sys.hash_info.algorithm, "siphash13") else: - self.assertIn(sys.hash_info.algorithm, {"fnv", "siphash24"}) + self.assertIn(sys.hash_info.algorithm, {"fnv", "siphash13", "siphash24"}) else: # PY_HASH_EXTERNAL self.assertEqual(algo, 0) diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-07-19-09-12.bpo-29410.bg5SYp.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-19-09-12.bpo-29410.bg5SYp.rst new file mode 100644 index 0000000000000..b08999e78d84a --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-07-19-09-12.bpo-29410.bg5SYp.rst @@ -0,0 +1 @@ +Add SipHash13 for string hash algorithm and use it by default. diff --git a/Python/pyhash.c b/Python/pyhash.c index f0c82356f1e26..d5ac9f83be61c 100644 --- a/Python/pyhash.c +++ b/Python/pyhash.c @@ -358,19 +358,72 @@ static PyHash_FuncDef PyHash_Func = {fnv, "fnv", 8 * SIZEOF_PY_HASH_T, # define ROTATE(x, b) (uint64_t)( ((x) << (b)) | ( (x) >> (64 - (b))) ) #endif -#define HALF_ROUND(a,b,c,d,s,t) \ - a += b; c += d; \ +#define HALF_ROUND(a,b,c,d,s,t) \ + a += b; c += d; \ b = ROTATE(b, s) ^ a; \ d = ROTATE(d, t) ^ c; \ a = ROTATE(a, 32); -#define DOUBLE_ROUND(v0,v1,v2,v3) \ - HALF_ROUND(v0,v1,v2,v3,13,16); \ - HALF_ROUND(v2,v1,v0,v3,17,21); \ - HALF_ROUND(v0,v1,v2,v3,13,16); \ +#define SINGLE_ROUND(v0,v1,v2,v3) \ + HALF_ROUND(v0,v1,v2,v3,13,16); \ HALF_ROUND(v2,v1,v0,v3,17,21); +#define DOUBLE_ROUND(v0,v1,v2,v3) \ + SINGLE_ROUND(v0,v1,v2,v3); \ + SINGLE_ROUND(v0,v1,v2,v3); + +static uint64_t +siphash13(uint64_t k0, uint64_t k1, const void *src, Py_ssize_t src_sz) { + uint64_t b = (uint64_t)src_sz << 56; + const uint8_t *in = (const uint8_t*)src; + + uint64_t v0 = k0 ^ 0x736f6d6570736575ULL; + uint64_t v1 = k1 ^ 0x646f72616e646f6dULL; + uint64_t v2 = k0 ^ 0x6c7967656e657261ULL; + uint64_t v3 = k1 ^ 0x7465646279746573ULL; + + uint64_t t; + uint8_t *pt; + + while (src_sz >= 8) { + uint64_t mi; + memcpy(&mi, in, sizeof(mi)); + mi = _le64toh(mi); + in += sizeof(mi); + src_sz -= sizeof(mi); + v3 ^= mi; + SINGLE_ROUND(v0,v1,v2,v3); + v0 ^= mi; + } + + t = 0; + pt = (uint8_t *)&t; + switch (src_sz) { + case 7: pt[6] = in[6]; /* fall through */ + case 6: pt[5] = in[5]; /* fall through */ + case 5: pt[4] = in[4]; /* fall through */ + case 4: memcpy(pt, in, sizeof(uint32_t)); break; + case 3: pt[2] = in[2]; /* fall through */ + case 2: pt[1] = in[1]; /* fall through */ + case 1: pt[0] = in[0]; /* fall through */ + } + b |= _le64toh(t); + + v3 ^= b; + SINGLE_ROUND(v0,v1,v2,v3); + v0 ^= b; + v2 ^= 0xff; + SINGLE_ROUND(v0,v1,v2,v3); + SINGLE_ROUND(v0,v1,v2,v3); + SINGLE_ROUND(v0,v1,v2,v3); + + /* modified */ + t = (v0 ^ v1) ^ (v2 ^ v3); + return t; +} + +#if Py_HASH_ALGORITHM == Py_HASH_SIPHASH24 static uint64_t siphash24(uint64_t k0, uint64_t k1, const void *src, Py_ssize_t src_sz) { uint64_t b = (uint64_t)src_sz << 56; @@ -419,14 +472,26 @@ siphash24(uint64_t k0, uint64_t k1, const void *src, Py_ssize_t src_sz) { t = (v0 ^ v1) ^ (v2 ^ v3); return t; } +#endif uint64_t _Py_KeyedHash(uint64_t key, const void *src, Py_ssize_t src_sz) { - return siphash24(key, 0, src, src_sz); + return siphash13(key, 0, src, src_sz); } +#if Py_HASH_ALGORITHM == Py_HASH_SIPHASH13 +static Py_hash_t +pysiphash(const void *src, Py_ssize_t src_sz) { + return (Py_hash_t)siphash13( + _le64toh(_Py_HashSecret.siphash.k0), _le64toh(_Py_HashSecret.siphash.k1), + src, src_sz); +} + +static PyHash_FuncDef PyHash_Func = {pysiphash, "siphash13", 64, 128}; +#endif + #if Py_HASH_ALGORITHM == Py_HASH_SIPHASH24 static Py_hash_t pysiphash(const void *src, Py_ssize_t src_sz) { diff --git a/configure b/configure index 3a6cf305171bc..15c7c54b09536 100755 --- a/configure +++ b/configure @@ -1561,9 +1561,9 @@ Optional Packages: --with-undefined-behavior-sanitizer enable UndefinedBehaviorSanitizer undefined behaviour detector, 'ubsan' (default is no) - --with-hash-algorithm=[fnv|siphash24] + --with-hash-algorithm=[fnv|siphash13|siphash24] select hash algorithm for use in Python/pyhash.c - (default is SipHash24) + (default is SipHash13) --with-tzpath= Select the default time zone search path for zoneinfo.TZPATH @@ -10431,6 +10431,10 @@ if test "${with_hash_algorithm+set}" = set; then : { $as_echo "$as_me:${as_lineno-$LINENO}: result: $withval" >&5 $as_echo "$withval" >&6; } case "$withval" in + siphash13) + $as_echo "#define Py_HASH_ALGORITHM 3" >>confdefs.h + + ;; siphash24) $as_echo "#define Py_HASH_ALGORITHM 1" >>confdefs.h diff --git a/configure.ac b/configure.ac index c7cb797e8f3a0..6c65b2914bf66 100644 --- a/configure.ac +++ b/configure.ac @@ -3036,16 +3036,19 @@ fi # str, bytes and memoryview hash algorithm AH_TEMPLATE(Py_HASH_ALGORITHM, [Define hash algorithm for str, bytes and memoryview. - SipHash24: 1, FNV: 2, externally defined: 0]) + SipHash24: 1, FNV: 2, SipHash13: 3, externally defined: 0]) AC_MSG_CHECKING(for --with-hash-algorithm) dnl quadrigraphs "@<:@" and "@:>@" produce "[" and "]" in the output AC_ARG_WITH(hash_algorithm, - AS_HELP_STRING([--with-hash-algorithm=@<:@fnv|siphash24@:>@], - [select hash algorithm for use in Python/pyhash.c (default is SipHash24)]), + AS_HELP_STRING([--with-hash-algorithm=@<:@fnv|siphash13|siphash24@:>@], + [select hash algorithm for use in Python/pyhash.c (default is SipHash13)]), [ AC_MSG_RESULT($withval) case "$withval" in + siphash13) + AC_DEFINE(Py_HASH_ALGORITHM, 3) + ;; siphash24) AC_DEFINE(Py_HASH_ALGORITHM, 1) ;; diff --git a/pyconfig.h.in b/pyconfig.h.in index 862c083604ee4..3231cd68e063f 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1439,7 +1439,7 @@ #undef Py_ENABLE_SHARED /* Define hash algorithm for str, bytes and memoryview. SipHash24: 1, FNV: 2, - externally defined: 0 */ + SipHash13: 3, externally defined: 0 */ #undef Py_HASH_ALGORITHM /* Define if you want to enable tracing references for debugging purpose */ From webhook-mailer at python.org Sun Oct 10 10:05:27 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Sun, 10 Oct 2021 14:05:27 -0000 Subject: [Python-checkins] Remove repeated 'the' in docs (GH-28852) Message-ID: https://github.com/python/cpython/commit/532403e7c64b079934e98f1dc29af7c603eb3db7 commit: 532403e7c64b079934e98f1dc29af7c603eb3db7 branch: main author: 180909 <734461790 at qq.com> committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-10T22:05:21+08:00 summary: Remove repeated 'the' in docs (GH-28852) files: M Misc/NEWS.d/3.10.0a7.rst M Misc/NEWS.d/3.11.0a1.rst diff --git a/Misc/NEWS.d/3.10.0a7.rst b/Misc/NEWS.d/3.10.0a7.rst index 6c32e60dc8a46..aa332631292a7 100644 --- a/Misc/NEWS.d/3.10.0a7.rst +++ b/Misc/NEWS.d/3.10.0a7.rst @@ -172,7 +172,7 @@ regression introduced in python3.7 .. nonce: Ns3a_F .. section: Core and Builtins -Tracing now has correct line numbers for attribute accesses when the the +Tracing now has correct line numbers for attribute accesses when the attribute is on a different line from the object. Improves debugging and profiling for multi-line method chains. diff --git a/Misc/NEWS.d/3.11.0a1.rst b/Misc/NEWS.d/3.11.0a1.rst index bdfdab078c090..e3d2acc499968 100644 --- a/Misc/NEWS.d/3.11.0a1.rst +++ b/Misc/NEWS.d/3.11.0a1.rst @@ -1196,7 +1196,7 @@ detected. Patch by Pablo Galindo .. section: Core and Builtins ``PyCodeObject`` gained ``co_fastlocalnames`` and ``co_fastlocalkinds`` as -the the authoritative source of fast locals info. Marshaled code objects +the authoritative source of fast locals info. Marshaled code objects have changed accordingly. .. @@ -4960,7 +4960,7 @@ trashcan macros accessed directly :c:type:`PyThreadState` members like ``_tstate->trash_delete_nesting``, whereas the :c:type:`PyThreadState` structure is opaque in the limited C API. -Exclude also the the ``PyTrash_UNWIND_LEVEL`` constant from the C API. +Exclude also the ``PyTrash_UNWIND_LEVEL`` constant from the C API. Patch by Victor Stinner. From webhook-mailer at python.org Sun Oct 10 10:13:00 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Sun, 10 Oct 2021 14:13:00 -0000 Subject: [Python-checkins] Fix class pattern docs to refer to class patterns (GH-28849) Message-ID: https://github.com/python/cpython/commit/0bcc5ade9bff086a0b24d71307fae0a891f4efd2 commit: 0bcc5ade9bff086a0b24d71307fae0a891f4efd2 branch: main author: Christophe Nanteuil <35002064+christopheNan at users.noreply.github.com> committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-10T22:12:51+08:00 summary: Fix class pattern docs to refer to class patterns (GH-28849) files: M Doc/reference/compound_stmts.rst diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 5936cdf5ffc30..3011a168ede42 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -1017,7 +1017,7 @@ A class pattern represents a class and its positional and keyword arguments The same keyword should not be repeated in class patterns. -The following is the logical flow for matching a mapping pattern against a +The following is the logical flow for matching a class pattern against a subject value: #. If ``name_or_attr`` is not an instance of the builtin :class:`type` , raise From webhook-mailer at python.org Sun Oct 10 10:23:17 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 10 Oct 2021 14:23:17 -0000 Subject: [Python-checkins] [3.10] bpo-45419: Fix interfaces on DegenerateFiles.Path (GH-28844) Message-ID: https://github.com/python/cpython/commit/14a483aa400dda8346ac474ce22e2ba8d8126dff commit: 14a483aa400dda8346ac474ce22e2ba8d8126dff branch: 3.10 author: Jason R. Coombs committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-10T07:23:06-07:00 summary: [3.10] bpo-45419: Fix interfaces on DegenerateFiles.Path (GH-28844) files: A Misc/NEWS.d/next/Library/2021-10-09-20-53-13.bpo-45419.CauCgt.rst M Lib/importlib/_adapters.py diff --git a/Lib/importlib/_adapters.py b/Lib/importlib/_adapters.py index eedde49dd03ad..e72edd10705c2 100644 --- a/Lib/importlib/_adapters.py +++ b/Lib/importlib/_adapters.py @@ -46,10 +46,11 @@ def is_dir(self): def joinpath(self, other): return DegenerateFiles.Path() + @property def name(self): return '' - def open(self): + def open(self, mode='rb', *args, **kwargs): raise ValueError() def __init__(self, spec): diff --git a/Misc/NEWS.d/next/Library/2021-10-09-20-53-13.bpo-45419.CauCgt.rst b/Misc/NEWS.d/next/Library/2021-10-09-20-53-13.bpo-45419.CauCgt.rst new file mode 100644 index 0000000000000..a901d7453819f --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-09-20-53-13.bpo-45419.CauCgt.rst @@ -0,0 +1 @@ +Correct interfaces on DegenerateFiles.Path. From webhook-mailer at python.org Sun Oct 10 10:35:31 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 10 Oct 2021 14:35:31 -0000 Subject: [Python-checkins] Fix class pattern docs to refer to class patterns (GH-28849) Message-ID: https://github.com/python/cpython/commit/3c270130774fa4d19fd5c2728b9309625da40b87 commit: 3c270130774fa4d19fd5c2728b9309625da40b87 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-10T07:35:23-07:00 summary: Fix class pattern docs to refer to class patterns (GH-28849) (cherry picked from commit 0bcc5ade9bff086a0b24d71307fae0a891f4efd2) Co-authored-by: Christophe Nanteuil <35002064+christopheNan at users.noreply.github.com> files: M Doc/reference/compound_stmts.rst diff --git a/Doc/reference/compound_stmts.rst b/Doc/reference/compound_stmts.rst index 41719be3dc986..3e5ef687922ff 100644 --- a/Doc/reference/compound_stmts.rst +++ b/Doc/reference/compound_stmts.rst @@ -1017,7 +1017,7 @@ A class pattern represents a class and its positional and keyword arguments The same keyword should not be repeated in class patterns. -The following is the logical flow for matching a mapping pattern against a +The following is the logical flow for matching a class pattern against a subject value: #. If ``name_or_attr`` is not an instance of the builtin :class:`type` , raise From webhook-mailer at python.org Sun Oct 10 11:15:28 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 10 Oct 2021 15:15:28 -0000 Subject: [Python-checkins] =?utf-8?q?bpo-45401=3A_Change_shouldRollover?= =?utf-8?q?=28=29_methods_to_only_rollover_regular_f=E2=80=A6_=28GH-28822?= =?utf-8?q?=29?= Message-ID: https://github.com/python/cpython/commit/62a667784ba7b84611ebd50fa8a1a464cde32235 commit: 62a667784ba7b84611ebd50fa8a1a464cde32235 branch: main author: Vinay Sajip committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-10T08:15:24-07:00 summary: bpo-45401: Change shouldRollover() methods to only rollover regular f? (GH-28822) ?iles. Also changed some historical return values from 1 -> True and 0 -> False. files: M Lib/logging/handlers.py M Lib/test/test_logging.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index f1a2e3b69986e..b613bec1c4270 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -187,14 +187,17 @@ def shouldRollover(self, record): Basically, see if the supplied record would cause the file to exceed the size limit we have. """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False if self.stream is None: # delay was set... self.stream = self._open() if self.maxBytes > 0: # are we rolling over? msg = "%s\n" % self.format(record) self.stream.seek(0, 2) #due to non-posix-compliant Windows feature if self.stream.tell() + len(msg) >= self.maxBytes: - return 1 - return 0 + return True + return False class TimedRotatingFileHandler(BaseRotatingHandler): """ @@ -345,10 +348,13 @@ def shouldRollover(self, record): record is not used, as we are just comparing times, but it is needed so the method signatures are the same """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False t = int(time.time()) if t >= self.rolloverAt: - return 1 - return 0 + return True + return False def getFilesToDelete(self): """ diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 7a80244047d0f..b5885b985afd3 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5216,6 +5216,13 @@ def test_should_not_rollover(self): self.fn, encoding="utf-8", maxBytes=0) self.assertFalse(rh.shouldRollover(None)) rh.close() + # bpo-45401 - test with special file + # We set maxBytes to 1 so that rollover would normally happen, except + # for the check for regular files + rh = logging.handlers.RotatingFileHandler( + os.devnull, encoding="utf-8", maxBytes=1) + self.assertFalse(rh.shouldRollover(self.next_rec())) + rh.close() def test_should_rollover(self): rh = logging.handlers.RotatingFileHandler(self.fn, encoding="utf-8", maxBytes=1) @@ -5310,6 +5317,14 @@ def rotator(source, dest): rh.close() class TimedRotatingFileHandlerTest(BaseFileTest): + def test_should_not_rollover(self): + # See bpo-45401. Should only ever rollover regular files + fh = logging.handlers.TimedRotatingFileHandler( + os.devnull, 'S', encoding="utf-8", backupCount=1) + time.sleep(1.1) # a little over a second ... + r = logging.makeLogRecord({'msg': 'testing - device file'}) + self.assertFalse(fh.shouldRollover(r)) + # other test methods added below def test_rollover(self): fh = logging.handlers.TimedRotatingFileHandler( From webhook-mailer at python.org Sun Oct 10 12:01:49 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 10 Oct 2021 16:01:49 -0000 Subject: [Python-checkins] bpo-45416: Fix use of asyncio.Condition() with explicit Lock objects (GH-28850) Message-ID: https://github.com/python/cpython/commit/1a7892414e654aa5c99efa31db767baba7f4a424 commit: 1a7892414e654aa5c99efa31db767baba7f4a424 branch: main author: Joongi Kim committer: serhiy-storchaka date: 2021-10-10T19:01:41+03:00 summary: bpo-45416: Fix use of asyncio.Condition() with explicit Lock objects (GH-28850) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst M Lib/asyncio/locks.py M Lib/test/test_asyncio/test_locks.py diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py index a7453fb1c7728..4fef64e3921e1 100644 --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -230,8 +230,6 @@ def __init__(self, lock=None, *, loop=mixins._marker): super().__init__(loop=loop) if lock is None: lock = Lock() - elif lock._loop is not self._get_loop(): - raise ValueError("loop argument must agree with lock") self._lock = lock # Export the lock's locked(), acquire() and release() methods. diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py index 441adeea8f8e0..b2492c1acfece 100644 --- a/Lib/test/test_asyncio/test_locks.py +++ b/Lib/test/test_asyncio/test_locks.py @@ -720,24 +720,68 @@ async def f(): self.loop.run_until_complete(f()) def test_explicit_lock(self): - lock = asyncio.Lock() - cond = asyncio.Condition(lock) + async def f(lock=None, cond=None): + if lock is None: + lock = asyncio.Lock() + if cond is None: + cond = asyncio.Condition(lock) + self.assertIs(cond._lock, lock) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with cond: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with lock: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) - self.assertIs(cond._lock, lock) - self.assertIs(cond._loop, lock._loop) + # All should work in the same way. + self.loop.run_until_complete(f()) + self.loop.run_until_complete(f(asyncio.Lock())) + lock = asyncio.Lock() + self.loop.run_until_complete(f(lock, asyncio.Condition(lock))) def test_ambiguous_loops(self): - loop = self.new_test_loop() + loop = asyncio.new_event_loop() self.addCleanup(loop.close) - lock = asyncio.Lock() - lock._loop = loop - - async def _create_condition(): - with self.assertRaises(ValueError): - asyncio.Condition(lock) - - self.loop.run_until_complete(_create_condition()) + async def wrong_loop_in_lock(): + with self.assertRaises(TypeError): + asyncio.Lock(loop=loop) # actively disallowed since 3.10 + lock = asyncio.Lock() + lock._loop = loop # use private API for testing + async with lock: + # acquired immediately via the fast-path + # without interaction with any event loop. + cond = asyncio.Condition(lock) + # cond.acquire() will trigger waiting on the lock + # and it will discover the event loop mismatch. + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.acquire() + + async def wrong_loop_in_cond(): + # Same analogy here with the condition's loop. + lock = asyncio.Lock() + async with lock: + with self.assertRaises(TypeError): + asyncio.Condition(lock, loop=loop) + cond = asyncio.Condition(lock) + cond._loop = loop + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.wait() + + self.loop.run_until_complete(wrong_loop_in_lock()) + self.loop.run_until_complete(wrong_loop_in_cond()) def test_timeout_in_block(self): loop = asyncio.new_event_loop() diff --git a/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst b/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst new file mode 100644 index 0000000000000..cf335d1bcd2c9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst @@ -0,0 +1,2 @@ +Fix use of :class:`asyncio.Condition` with explicit :class:`asyncio.Lock` objects, which was a regression due to removal of explicit loop arguments. +Patch by Joongi Kim. \ No newline at end of file From webhook-mailer at python.org Sun Oct 10 12:25:22 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 10 Oct 2021 16:25:22 -0000 Subject: [Python-checkins] bpo-45416: Fix use of asyncio.Condition() with explicit Lock objects (GH-28850) Message-ID: https://github.com/python/cpython/commit/164dddf5f8c9c6b93f32c9f79b4301fc804576e9 commit: 164dddf5f8c9c6b93f32c9f79b4301fc804576e9 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-10T09:25:14-07:00 summary: bpo-45416: Fix use of asyncio.Condition() with explicit Lock objects (GH-28850) Co-authored-by: Serhiy Storchaka (cherry picked from commit 1a7892414e654aa5c99efa31db767baba7f4a424) Co-authored-by: Joongi Kim files: A Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst M Lib/asyncio/locks.py M Lib/test/test_asyncio/test_locks.py diff --git a/Lib/asyncio/locks.py b/Lib/asyncio/locks.py index a7453fb1c7728..4fef64e3921e1 100644 --- a/Lib/asyncio/locks.py +++ b/Lib/asyncio/locks.py @@ -230,8 +230,6 @@ def __init__(self, lock=None, *, loop=mixins._marker): super().__init__(loop=loop) if lock is None: lock = Lock() - elif lock._loop is not self._get_loop(): - raise ValueError("loop argument must agree with lock") self._lock = lock # Export the lock's locked(), acquire() and release() methods. diff --git a/Lib/test/test_asyncio/test_locks.py b/Lib/test/test_asyncio/test_locks.py index 6194cd06176da..623db5fda6437 100644 --- a/Lib/test/test_asyncio/test_locks.py +++ b/Lib/test/test_asyncio/test_locks.py @@ -724,24 +724,68 @@ async def f(): self.loop.run_until_complete(f()) def test_explicit_lock(self): - lock = asyncio.Lock() - cond = asyncio.Condition(lock) + async def f(lock=None, cond=None): + if lock is None: + lock = asyncio.Lock() + if cond is None: + cond = asyncio.Condition(lock) + self.assertIs(cond._lock, lock) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with cond: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) + async with lock: + self.assertTrue(lock.locked()) + self.assertTrue(cond.locked()) + self.assertFalse(lock.locked()) + self.assertFalse(cond.locked()) - self.assertIs(cond._lock, lock) - self.assertIs(cond._loop, lock._loop) + # All should work in the same way. + self.loop.run_until_complete(f()) + self.loop.run_until_complete(f(asyncio.Lock())) + lock = asyncio.Lock() + self.loop.run_until_complete(f(lock, asyncio.Condition(lock))) def test_ambiguous_loops(self): - loop = self.new_test_loop() + loop = asyncio.new_event_loop() self.addCleanup(loop.close) - lock = asyncio.Lock() - lock._loop = loop + async def wrong_loop_in_lock(): + with self.assertRaises(TypeError): + asyncio.Lock(loop=loop) # actively disallowed since 3.10 + lock = asyncio.Lock() + lock._loop = loop # use private API for testing + async with lock: + # acquired immediately via the fast-path + # without interaction with any event loop. + cond = asyncio.Condition(lock) + # cond.acquire() will trigger waiting on the lock + # and it will discover the event loop mismatch. + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.acquire() - async def _create_condition(): - with self.assertRaises(ValueError): - asyncio.Condition(lock) + async def wrong_loop_in_cond(): + # Same analogy here with the condition's loop. + lock = asyncio.Lock() + async with lock: + with self.assertRaises(TypeError): + asyncio.Condition(lock, loop=loop) + cond = asyncio.Condition(lock) + cond._loop = loop + with self.assertRaisesRegex( + RuntimeError, + "is bound to a different event loop", + ): + await cond.wait() - self.loop.run_until_complete(_create_condition()) + self.loop.run_until_complete(wrong_loop_in_lock()) + self.loop.run_until_complete(wrong_loop_in_cond()) def test_timeout_in_block(self): loop = asyncio.new_event_loop() diff --git a/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst b/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst new file mode 100644 index 0000000000000..cf335d1bcd2c9 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-10-09-42-34.bpo-45416.n35O0_.rst @@ -0,0 +1,2 @@ +Fix use of :class:`asyncio.Condition` with explicit :class:`asyncio.Lock` objects, which was a regression due to removal of explicit loop arguments. +Patch by Joongi Kim. \ No newline at end of file From webhook-mailer at python.org Sun Oct 10 17:43:48 2021 From: webhook-mailer at python.org (Mariatta) Date: Sun, 10 Oct 2021 21:43:48 -0000 Subject: [Python-checkins] Fix the "Finding all Adverbs" example (GH-21420) (#28839) Message-ID: https://github.com/python/cpython/commit/aff69c34b04390488ff8a364894f9e0c25fc4435 commit: aff69c34b04390488ff8a364894f9e0c25fc4435 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-10T14:43:38-07:00 summary: Fix the "Finding all Adverbs" example (GH-21420) (#28839) Co-authored-by: Serhiy Storchaka (cherry picked from commit dbd62e74dadda7868f1c0d497414c8f7e4c0b12b) Co-authored-by: Rim Chatti files: M Doc/library/re.rst diff --git a/Doc/library/re.rst b/Doc/library/re.rst index ff7687cc936ec..b12ce4b9744f9 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1572,7 +1572,7 @@ find all of the adverbs in some text, they might use :func:`findall` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> re.findall(r"\w+ly", text) + >>> re.findall(r"\w+ly\b", text) ['carefully', 'quickly'] @@ -1586,7 +1586,7 @@ a writer wanted to find all of the adverbs *and their positions* in some text, they would use :func:`finditer` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> for m in re.finditer(r"\w+ly", text): + >>> for m in re.finditer(r"\w+ly\b", text): ... print('%02d-%02d: %s' % (m.start(), m.end(), m.group(0))) 07-16: carefully 40-47: quickly From webhook-mailer at python.org Sun Oct 10 17:44:07 2021 From: webhook-mailer at python.org (Mariatta) Date: Sun, 10 Oct 2021 21:44:07 -0000 Subject: [Python-checkins] Fix the "Finding all Adverbs" example (GH-21420) (#28840) Message-ID: https://github.com/python/cpython/commit/5f44bb28fd9e98d681adff7e3329d6a863d3d5cd commit: 5f44bb28fd9e98d681adff7e3329d6a863d3d5cd branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-10T14:43:58-07:00 summary: Fix the "Finding all Adverbs" example (GH-21420) (#28840) Co-authored-by: Serhiy Storchaka (cherry picked from commit dbd62e74dadda7868f1c0d497414c8f7e4c0b12b) Co-authored-by: Rim Chatti files: M Doc/library/re.rst diff --git a/Doc/library/re.rst b/Doc/library/re.rst index ff7687cc936ec..b12ce4b9744f9 100644 --- a/Doc/library/re.rst +++ b/Doc/library/re.rst @@ -1572,7 +1572,7 @@ find all of the adverbs in some text, they might use :func:`findall` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> re.findall(r"\w+ly", text) + >>> re.findall(r"\w+ly\b", text) ['carefully', 'quickly'] @@ -1586,7 +1586,7 @@ a writer wanted to find all of the adverbs *and their positions* in some text, they would use :func:`finditer` in the following manner:: >>> text = "He was carefully disguised but captured quickly by police." - >>> for m in re.finditer(r"\w+ly", text): + >>> for m in re.finditer(r"\w+ly\b", text): ... print('%02d-%02d: %s' % (m.start(), m.end(), m.group(0))) 07-16: carefully 40-47: quickly From webhook-mailer at python.org Mon Oct 11 04:54:49 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 11 Oct 2021 08:54:49 -0000 Subject: [Python-checkins] bpo-45401: Fix a resource warning in test_logging (GH-28864) Message-ID: https://github.com/python/cpython/commit/15188b115a2da815556053372c912a81a74be43b commit: 15188b115a2da815556053372c912a81a74be43b branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-11T11:54:44+03:00 summary: bpo-45401: Fix a resource warning in test_logging (GH-28864) files: M Lib/test/test_logging.py diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index b5885b985afd3..85b6e5f392111 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5324,6 +5324,7 @@ def test_should_not_rollover(self): time.sleep(1.1) # a little over a second ... r = logging.makeLogRecord({'msg': 'testing - device file'}) self.assertFalse(fh.shouldRollover(r)) + fh.close() # other test methods added below def test_rollover(self): From webhook-mailer at python.org Mon Oct 11 04:57:32 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 11 Oct 2021 08:57:32 -0000 Subject: [Python-checkins] Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) Message-ID: https://github.com/python/cpython/commit/9883ca498d654a4792d530bd8d6d64fef4dc971c commit: 9883ca498d654a4792d530bd8d6d64fef4dc971c branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-11T11:57:27+03:00 summary: Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 1847eba74aef4..27ffc64284098 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -121,7 +121,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) if (PySys_Audit("import", "OOOOO", name_unicode, path, Py_None, Py_None, Py_None) < 0) { - return NULL; + goto error; } #ifdef MS_WINDOWS From webhook-mailer at python.org Mon Oct 11 05:22:34 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 09:22:34 -0000 Subject: [Python-checkins] Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) Message-ID: https://github.com/python/cpython/commit/3a58d6062060f9fcc1f4f9c43358769305bafef8 commit: 3a58d6062060f9fcc1f4f9c43358769305bafef8 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T02:22:29-07:00 summary: Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) (cherry picked from commit 9883ca498d654a4792d530bd8d6d64fef4dc971c) Co-authored-by: Serhiy Storchaka files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 1847eba74aef4..27ffc64284098 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -121,7 +121,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) if (PySys_Audit("import", "OOOOO", name_unicode, path, Py_None, Py_None, Py_None) < 0) { - return NULL; + goto error; } #ifdef MS_WINDOWS From webhook-mailer at python.org Mon Oct 11 05:43:26 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 09:43:26 -0000 Subject: [Python-checkins] Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) Message-ID: https://github.com/python/cpython/commit/d57d33c2342f8ebaf9a91fe991dbfc148340e811 commit: d57d33c2342f8ebaf9a91fe991dbfc148340e811 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T02:43:09-07:00 summary: Fix a leak in _PyImport_LoadDynamicModuleWithSpec() after failing PySys_Audit() (GH-28862) (cherry picked from commit 9883ca498d654a4792d530bd8d6d64fef4dc971c) Co-authored-by: Serhiy Storchaka files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index fbeb9fb75403e..b197dfeaef805 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -121,7 +121,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) if (PySys_Audit("import", "OOOOO", name_unicode, path, Py_None, Py_None, Py_None) < 0) { - return NULL; + goto error; } #ifdef MS_WINDOWS From webhook-mailer at python.org Mon Oct 11 06:01:18 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 10:01:18 -0000 Subject: [Python-checkins] =?utf-8?q?bpo-45401=3A_Change_shouldRollover?= =?utf-8?q?=28=29_methods_to_only_rollover_regular_f=E2=80=A6_=28GH-28822?= =?utf-8?b?KSAoIzI4ODY2KQ==?= Message-ID: https://github.com/python/cpython/commit/ac421c348bf422f9a0d85fe0a1de3fa3f4886650 commit: ac421c348bf422f9a0d85fe0a1de3fa3f4886650 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vstinner date: 2021-10-11T12:01:09+02:00 summary: bpo-45401: Change shouldRollover() methods to only rollover regular f? (GH-28822) (#28866) ?iles. Also changed some historical return values from 1 -> True and 0 -> False. (cherry picked from commit 62a667784ba7b84611ebd50fa8a1a464cde32235) Co-authored-by: Vinay Sajip Co-authored-by: Vinay Sajip files: M Lib/logging/handlers.py M Lib/test/test_logging.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 74b67602fa94f..7f1f10551c304 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -185,14 +185,17 @@ def shouldRollover(self, record): Basically, see if the supplied record would cause the file to exceed the size limit we have. """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False if self.stream is None: # delay was set... self.stream = self._open() if self.maxBytes > 0: # are we rolling over? msg = "%s\n" % self.format(record) self.stream.seek(0, 2) #due to non-posix-compliant Windows feature if self.stream.tell() + len(msg) >= self.maxBytes: - return 1 - return 0 + return True + return False class TimedRotatingFileHandler(BaseRotatingHandler): """ @@ -342,10 +345,13 @@ def shouldRollover(self, record): record is not used, as we are just comparing times, but it is needed so the method signatures are the same """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False t = int(time.time()) if t >= self.rolloverAt: - return 1 - return 0 + return True + return False def getFilesToDelete(self): """ diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 08fb79506c5ad..8202879df067c 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5066,6 +5066,13 @@ def test_should_not_rollover(self): rh = logging.handlers.RotatingFileHandler(self.fn, maxBytes=0) self.assertFalse(rh.shouldRollover(None)) rh.close() + # bpo-45401 - test with special file + # We set maxBytes to 1 so that rollover would normally happen, except + # for the check for regular files + rh = logging.handlers.RotatingFileHandler( + os.devnull, encoding="utf-8", maxBytes=1) + self.assertFalse(rh.shouldRollover(self.next_rec())) + rh.close() def test_should_rollover(self): rh = logging.handlers.RotatingFileHandler(self.fn, maxBytes=1) @@ -5160,6 +5167,14 @@ def rotator(source, dest): rh.close() class TimedRotatingFileHandlerTest(BaseFileTest): + def test_should_not_rollover(self): + # See bpo-45401. Should only ever rollover regular files + fh = logging.handlers.TimedRotatingFileHandler( + os.devnull, 'S', encoding="utf-8", backupCount=1) + time.sleep(1.1) # a little over a second ... + r = logging.makeLogRecord({'msg': 'testing - device file'}) + self.assertFalse(fh.shouldRollover(r)) + # other test methods added below def test_rollover(self): fh = logging.handlers.TimedRotatingFileHandler(self.fn, 'S', From webhook-mailer at python.org Mon Oct 11 06:01:18 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 10:01:18 -0000 Subject: [Python-checkins] =?utf-8?q?bpo-45401=3A_Change_shouldRollover?= =?utf-8?q?=28=29_methods_to_only_rollover_regular_f=E2=80=A6_=28GH-28822?= =?utf-8?b?KSAoIzI4ODY3KQ==?= Message-ID: https://github.com/python/cpython/commit/5aca34f17c4baf8e4882a7e8a827cff06ac6ef25 commit: 5aca34f17c4baf8e4882a7e8a827cff06ac6ef25 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vstinner date: 2021-10-11T12:01:12+02:00 summary: bpo-45401: Change shouldRollover() methods to only rollover regular f? (GH-28822) (#28867) ?iles. Also changed some historical return values from 1 -> True and 0 -> False. (cherry picked from commit 62a667784ba7b84611ebd50fa8a1a464cde32235) Co-authored-by: Vinay Sajip Co-authored-by: Vinay Sajip files: M Lib/logging/handlers.py M Lib/test/test_logging.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 4cabc0d740c29..4dcbe4530fcfc 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -187,14 +187,17 @@ def shouldRollover(self, record): Basically, see if the supplied record would cause the file to exceed the size limit we have. """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False if self.stream is None: # delay was set... self.stream = self._open() if self.maxBytes > 0: # are we rolling over? msg = "%s\n" % self.format(record) self.stream.seek(0, 2) #due to non-posix-compliant Windows feature if self.stream.tell() + len(msg) >= self.maxBytes: - return 1 - return 0 + return True + return False class TimedRotatingFileHandler(BaseRotatingHandler): """ @@ -345,10 +348,13 @@ def shouldRollover(self, record): record is not used, as we are just comparing times, but it is needed so the method signatures are the same """ + # See bpo-45401: Never rollover anything other than regular files + if os.path.exists(self.baseFilename) and not os.path.isfile(self.baseFilename): + return False t = int(time.time()) if t >= self.rolloverAt: - return 1 - return 0 + return True + return False def getFilesToDelete(self): """ diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 8356e6be24baf..1a5f8b60f5ccb 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5190,6 +5190,13 @@ def test_should_not_rollover(self): self.fn, encoding="utf-8", maxBytes=0) self.assertFalse(rh.shouldRollover(None)) rh.close() + # bpo-45401 - test with special file + # We set maxBytes to 1 so that rollover would normally happen, except + # for the check for regular files + rh = logging.handlers.RotatingFileHandler( + os.devnull, encoding="utf-8", maxBytes=1) + self.assertFalse(rh.shouldRollover(self.next_rec())) + rh.close() def test_should_rollover(self): rh = logging.handlers.RotatingFileHandler(self.fn, encoding="utf-8", maxBytes=1) @@ -5284,6 +5291,14 @@ def rotator(source, dest): rh.close() class TimedRotatingFileHandlerTest(BaseFileTest): + def test_should_not_rollover(self): + # See bpo-45401. Should only ever rollover regular files + fh = logging.handlers.TimedRotatingFileHandler( + os.devnull, 'S', encoding="utf-8", backupCount=1) + time.sleep(1.1) # a little over a second ... + r = logging.makeLogRecord({'msg': 'testing - device file'}) + self.assertFalse(fh.shouldRollover(r)) + # other test methods added below def test_rollover(self): fh = logging.handlers.TimedRotatingFileHandler( From webhook-mailer at python.org Mon Oct 11 06:34:32 2021 From: webhook-mailer at python.org (markshannon) Date: Mon, 11 Oct 2021 10:34:32 -0000 Subject: [Python-checkins] Restore PEP 523 functionality. (GH-28871) Message-ID: https://github.com/python/cpython/commit/fcb3d2ff633f3e9a826888d8b29c305a3959ff54 commit: fcb3d2ff633f3e9a826888d8b29c305a3959ff54 branch: main author: Mark Shannon committer: markshannon date: 2021-10-11T11:34:02+01:00 summary: Restore PEP 523 functionality. (GH-28871) files: M Include/internal/pycore_ceval.h M Python/ceval.c M Python/pystate.c diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h index 66ddc991a9b11..f2acf10df8c23 100644 --- a/Include/internal/pycore_ceval.h +++ b/Include/internal/pycore_ceval.h @@ -43,6 +43,9 @@ extern PyObject *_PyEval_BuiltinsFromGlobals( static inline PyObject* _PyEval_EvalFrame(PyThreadState *tstate, struct _interpreter_frame *frame, int throwflag) { + if (tstate->interp->eval_frame == NULL) { + return _PyEval_EvalFrameDefault(tstate, frame, throwflag); + } return tstate->interp->eval_frame(tstate, frame, throwflag); } diff --git a/Python/ceval.c b/Python/ceval.c index df435bf3d7681..a63b80395dcfc 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4612,7 +4612,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr // Check if the call can be inlined or not PyObject *function = PEEK(oparg + 1); - if (Py_TYPE(function) == &PyFunction_Type) { + if (Py_TYPE(function) == &PyFunction_Type && tstate->interp->eval_frame == NULL) { int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(function))->co_flags; PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); int is_generator = code_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR); @@ -4630,7 +4630,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } STACK_SHRINK(oparg + 1); - assert(tstate->interp->eval_frame != NULL); // The frame has stolen all the arguments from the stack, // so there is no need to clean them up. Py_DECREF(function); @@ -5687,7 +5686,7 @@ _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, if (steal_args) { // If we failed to initialize locals, make sure the caller still own all the // arguments. Notice that we only need to increase the reference count of the - // *valid* arguments (i.e. the ones that fit into the frame). + // *valid* arguments (i.e. the ones that fit into the frame). PyCodeObject *co = (PyCodeObject*)con->fc_code; const size_t total_args = co->co_argcount + co->co_kwonlyargcount; for (size_t i = 0; i < Py_MIN(argcount, total_args); i++) { @@ -5734,7 +5733,6 @@ _PyEval_Vector(PyThreadState *tstate, PyFrameConstructor *con, if (frame == NULL) { return NULL; } - assert (tstate->interp->eval_frame != NULL); PyObject *retval = _PyEval_EvalFrame(tstate, frame, 0); assert(_PyFrame_GetStackPointer(frame) == _PyFrame_Stackbase(frame)); if (_PyEvalFrameClearAndPop(tstate, frame)) { diff --git a/Python/pystate.c b/Python/pystate.c index 45272142e4173..f02de926c6f02 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -230,7 +230,7 @@ PyInterpreterState_New(void) PyConfig_InitPythonConfig(&interp->config); _PyType_InitCache(interp); - interp->eval_frame = _PyEval_EvalFrameDefault; + interp->eval_frame = NULL; #ifdef HAVE_DLOPEN #if HAVE_DECL_RTLD_NOW interp->dlopenflags = RTLD_NOW; @@ -1973,6 +1973,9 @@ _register_builtins_for_crossinterpreter_data(struct _xidregistry *xidregistry) _PyFrameEvalFunction _PyInterpreterState_GetEvalFrameFunc(PyInterpreterState *interp) { + if (interp->eval_frame == NULL) { + return _PyEval_EvalFrameDefault; + } return interp->eval_frame; } @@ -1981,7 +1984,12 @@ void _PyInterpreterState_SetEvalFrameFunc(PyInterpreterState *interp, _PyFrameEvalFunction eval_frame) { - interp->eval_frame = eval_frame; + if (eval_frame == _PyEval_EvalFrameDefault) { + interp->eval_frame = NULL; + } + else { + interp->eval_frame = eval_frame; + } } From webhook-mailer at python.org Mon Oct 11 06:42:10 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Mon, 11 Oct 2021 10:42:10 -0000 Subject: [Python-checkins] bpo-42253: Update xml.dom.minidom.rst (GH-23126) Message-ID: https://github.com/python/cpython/commit/c7e81fcf9548ab6a0a4828d6f2db9ece9d204826 commit: c7e81fcf9548ab6a0a4828d6f2db9ece9d204826 branch: main author: Jens Diemer committer: serhiy-storchaka date: 2021-10-11T13:42:05+03:00 summary: bpo-42253: Update xml.dom.minidom.rst (GH-23126) Document that the "standalone" parameter was added in Python 3.9. Co-authored-by: Serhiy Storchaka files: M Doc/library/xml.dom.minidom.rst diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index d3a5f872204a3..20984b98b1778 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -156,6 +156,9 @@ module documentation. This section lists the differences between the API and The :meth:`writexml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toxml(encoding=None, standalone=None) Return a string or byte string containing the XML represented by @@ -174,6 +177,9 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None, \ standalone=None) @@ -190,6 +196,8 @@ module documentation. This section lists the differences between the API and The :meth:`toprettyxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. .. _dom-example: From webhook-mailer at python.org Mon Oct 11 07:05:43 2021 From: webhook-mailer at python.org (JulienPalard) Date: Mon, 11 Oct 2021 11:05:43 -0000 Subject: [Python-checkins] bpo-45411: Update mimetypes.py (GH-28792) Message-ID: https://github.com/python/cpython/commit/d74da9e140441135a4eddaef9a37f00f32579038 commit: d74da9e140441135a4eddaef9a37f00f32579038 branch: main author: Josephine-Marie <75948248+Josephine-Marie at users.noreply.github.com> committer: JulienPalard date: 2021-10-11T13:05:28+02:00 summary: bpo-45411: Update mimetypes.py (GH-28792) .vtt and .srt files are common subtitle files, used by browsers. files: A Misc/NEWS.d/next/Library/2021-10-08-11-29-29.bpo-45411.4jR--U.rst M Lib/mimetypes.py diff --git a/Lib/mimetypes.py b/Lib/mimetypes.py index 3ba91771ed03d..e14d0719bed48 100644 --- a/Lib/mimetypes.py +++ b/Lib/mimetypes.py @@ -552,8 +552,10 @@ def _default_mime_types(): '.h' : 'text/plain', '.ksh' : 'text/plain', '.pl' : 'text/plain', + '.srt' : 'text/plain', '.rtx' : 'text/richtext', '.tsv' : 'text/tab-separated-values', + '.vtt' : 'text/vtt', '.py' : 'text/x-python', '.etx' : 'text/x-setext', '.sgm' : 'text/x-sgml', diff --git a/Misc/NEWS.d/next/Library/2021-10-08-11-29-29.bpo-45411.4jR--U.rst b/Misc/NEWS.d/next/Library/2021-10-08-11-29-29.bpo-45411.4jR--U.rst new file mode 100644 index 0000000000000..5f774dcda9849 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-08-11-29-29.bpo-45411.4jR--U.rst @@ -0,0 +1 @@ +Add extensions for files containing subtitles - .srt & .vtt - to the mimetypes.py module. From webhook-mailer at python.org Mon Oct 11 07:08:20 2021 From: webhook-mailer at python.org (corona10) Date: Mon, 11 Oct 2021 11:08:20 -0000 Subject: [Python-checkins] bpo-20028: Empty escapechar/quotechar is not allowed for csv.Dialect (GH-28833) Message-ID: https://github.com/python/cpython/commit/ab62051152cb24470056ffaeb9107c8b4311375e commit: ab62051152cb24470056ffaeb9107c8b4311375e branch: main author: Dong-hee Na committer: corona10 date: 2021-10-11T20:08:15+09:00 summary: bpo-20028: Empty escapechar/quotechar is not allowed for csv.Dialect (GH-28833) files: A Misc/NEWS.d/next/Library/2021-10-10-00-25-36.bpo-20028.bPx4Z8.rst M Doc/library/csv.rst M Lib/test/test_csv.py M Modules/_csv.c diff --git a/Doc/library/csv.rst b/Doc/library/csv.rst index 899ce0225ce7f..3a7817cfdfad8 100644 --- a/Doc/library/csv.rst +++ b/Doc/library/csv.rst @@ -383,6 +383,8 @@ Dialects support the following attributes: :const:`False`. On reading, the *escapechar* removes any special meaning from the following character. It defaults to :const:`None`, which disables escaping. + .. versionchanged:: 3.11 + An empty *escapechar* is not allowed. .. attribute:: Dialect.lineterminator @@ -402,6 +404,8 @@ Dialects support the following attributes: as the *delimiter* or *quotechar*, or which contain new-line characters. It defaults to ``'"'``. + .. versionchanged:: 3.11 + An empty *quotechar* is not allowed. .. attribute:: Dialect.quoting diff --git a/Lib/test/test_csv.py b/Lib/test/test_csv.py index fb27ea396e04d..95a19dd46cb4f 100644 --- a/Lib/test/test_csv.py +++ b/Lib/test/test_csv.py @@ -44,6 +44,8 @@ def _test_arg_valid(self, ctor, arg): quoting=csv.QUOTE_ALL, quotechar='') self.assertRaises(TypeError, ctor, arg, quoting=csv.QUOTE_ALL, quotechar=None) + self.assertRaises(TypeError, ctor, arg, + quoting=csv.QUOTE_NONE, quotechar='') def test_reader_arg_valid(self): self._test_arg_valid(csv.reader, []) @@ -342,7 +344,6 @@ def test_read_escape(self): self._read_test(['a,^b,c'], [['a', 'b', 'c']], escapechar='^') self._read_test(['a,\0b,c'], [['a', 'b', 'c']], escapechar='\0') self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar=None) - self._read_test(['a,\\b,c'], [['a', '\\b', 'c']], escapechar='') self._read_test(['a,\\b,c'], [['a', '\\b', 'c']]) def test_read_quoting(self): @@ -913,6 +914,12 @@ class mydialect(csv.Dialect): self.assertEqual(d.quotechar, '"') self.assertTrue(d.doublequote) + mydialect.quotechar = "" + with self.assertRaises(csv.Error) as cm: + mydialect() + self.assertEqual(str(cm.exception), + '"quotechar" must be a 1-character string') + mydialect.quotechar = "''" with self.assertRaises(csv.Error) as cm: mydialect() @@ -977,6 +984,10 @@ class mydialect(csv.Dialect): d = mydialect() self.assertEqual(d.escapechar, "\\") + mydialect.escapechar = "" + with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'): + mydialect() + mydialect.escapechar = "**" with self.assertRaisesRegex(csv.Error, '"escapechar" must be a 1-character string'): mydialect() diff --git a/Misc/NEWS.d/next/Library/2021-10-10-00-25-36.bpo-20028.bPx4Z8.rst b/Misc/NEWS.d/next/Library/2021-10-10-00-25-36.bpo-20028.bPx4Z8.rst new file mode 100644 index 0000000000000..9db15bc39e7ca --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-10-00-25-36.bpo-20028.bPx4Z8.rst @@ -0,0 +1,2 @@ +Empty escapechar/quotechar is not allowed when initializing +:class:`csv.Dialect`. Patch by Vajrasky Kok and Dong-hee Na. diff --git a/Modules/_csv.c b/Modules/_csv.c index 469c1a15c340c..1c2f504ea5c09 100644 --- a/Modules/_csv.c +++ b/Modules/_csv.c @@ -250,16 +250,14 @@ _set_char_or_none(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt if (len < 0) { return -1; } - if (len > 1) { + if (len != 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", name); return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - else if (len > 0) { - *target = PyUnicode_READ_CHAR(src, 0); - } + *target = PyUnicode_READ_CHAR(src, 0); } } return 0; @@ -272,7 +270,6 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) *target = dflt; } else { - *target = NOT_SET; if (!PyUnicode_Check(src)) { PyErr_Format(PyExc_TypeError, "\"%s\" must be string, not %.200s", name, @@ -283,16 +280,14 @@ _set_char(const char *name, Py_UCS4 *target, PyObject *src, Py_UCS4 dflt) if (len < 0) { return -1; } - if (len > 1) { + if (len != 1) { PyErr_Format(PyExc_TypeError, "\"%s\" must be a 1-character string", name); return -1; } /* PyUnicode_READY() is called in PyUnicode_GetLength() */ - else if (len > 0) { - *target = PyUnicode_READ_CHAR(src, 0); - } + *target = PyUnicode_READ_CHAR(src, 0); } return 0; } From webhook-mailer at python.org Mon Oct 11 07:08:42 2021 From: webhook-mailer at python.org (corona10) Date: Mon, 11 Oct 2021 11:08:42 -0000 Subject: [Python-checkins] Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) Message-ID: https://github.com/python/cpython/commit/560a79f94e94de66a18f2a5e4194c2fe51e2adf1 commit: 560a79f94e94de66a18f2a5e4194c2fe51e2adf1 branch: main author: Dong-hee Na committer: corona10 date: 2021-10-11T20:08:38+09:00 summary: Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 27ffc64284098..3d9cd1ac8673c 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -42,6 +42,9 @@ get_encoded_name(PyObject *name, const char **hook_prefix) { /* Get the short name (substring after last dot) */ name_len = PyUnicode_GetLength(name); + if (name_len < 0) { + return NULL; + } lastdot = PyUnicode_FindChar(name, '.', 0, name_len, -1); if (lastdot < -1) { return NULL; From webhook-mailer at python.org Mon Oct 11 07:41:00 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 11:41:00 -0000 Subject: [Python-checkins] Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) Message-ID: https://github.com/python/cpython/commit/56825b697e0831bf431fd83c59dc5c4d35292560 commit: 56825b697e0831bf431fd83c59dc5c4d35292560 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T04:40:43-07:00 summary: Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) (cherry picked from commit 560a79f94e94de66a18f2a5e4194c2fe51e2adf1) Co-authored-by: Dong-hee Na files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 27ffc64284098..3d9cd1ac8673c 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -42,6 +42,9 @@ get_encoded_name(PyObject *name, const char **hook_prefix) { /* Get the short name (substring after last dot) */ name_len = PyUnicode_GetLength(name); + if (name_len < 0) { + return NULL; + } lastdot = PyUnicode_FindChar(name, '.', 0, name_len, -1); if (lastdot < -1) { return NULL; From webhook-mailer at python.org Mon Oct 11 07:41:02 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 11:41:02 -0000 Subject: [Python-checkins] Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) Message-ID: https://github.com/python/cpython/commit/7d1ae89b846c6965f7e0de1cc01a89db90c2667a commit: 7d1ae89b846c6965f7e0de1cc01a89db90c2667a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T04:40:57-07:00 summary: Handle error when PyUnicode_GetLength returns a negative value. (GH-28859) (cherry picked from commit 560a79f94e94de66a18f2a5e4194c2fe51e2adf1) Co-authored-by: Dong-hee Na files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index b197dfeaef805..7f600d4a7aaf8 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -42,6 +42,9 @@ get_encoded_name(PyObject *name, const char **hook_prefix) { /* Get the short name (substring after last dot) */ name_len = PyUnicode_GetLength(name); + if (name_len < 0) { + return NULL; + } lastdot = PyUnicode_FindChar(name, '.', 0, name_len, -1); if (lastdot < -1) { return NULL; From webhook-mailer at python.org Mon Oct 11 14:54:53 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 18:54:53 -0000 Subject: [Python-checkins] bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) Message-ID: https://github.com/python/cpython/commit/659812b451aefe1f0e5f83540296519a5fb8f313 commit: 659812b451aefe1f0e5f83540296519a5fb8f313 branch: main author: Olaf van der Spek committer: vstinner date: 2021-10-11T20:54:44+02:00 summary: bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index b3e229c24f07d..95a8e4649beed 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -395,8 +395,8 @@ TCP echo server using the :func:`asyncio.start_server` function:: server = await asyncio.start_server( handle_echo, '127.0.0.1', 8888) - addr = server.sockets[0].getsockname() - print(f'Serving on {addr}') + addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets) + print(f'Serving on {addrs}') async with server: await server.serve_forever() From webhook-mailer at python.org Mon Oct 11 15:00:30 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 19:00:30 -0000 Subject: [Python-checkins] bpo-45412: Remove Py_SET_ERRNO_ON_MATH_ERROR() macro (GH-28820) Message-ID: https://github.com/python/cpython/commit/2f92e2a590f0e5d2d3093549f5af9a4a1889eb5a commit: 2f92e2a590f0e5d2d3093549f5af9a4a1889eb5a branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T21:00:25+02:00 summary: bpo-45412: Remove Py_SET_ERRNO_ON_MATH_ERROR() macro (GH-28820) Remove the following math macros using the errno variable: * Py_ADJUST_ERANGE1() * Py_ADJUST_ERANGE2() * Py_OVERFLOWED() * Py_SET_ERANGE_IF_OVERFLOW() * Py_SET_ERRNO_ON_MATH_ERROR() Create pycore_pymath.h internal header file. Rename Py_ADJUST_ERANGE1() and Py_ADJUST_ERANGE2() to _Py_ADJUST_ERANGE1() and _Py_ADJUST_ERANGE2(), and convert these macros to static inline functions. Move the following macros to pycore_pymath.h: * _Py_IntegralTypeSigned() * _Py_IntegralTypeMax() * _Py_IntegralTypeMin() * _Py_InIntegralTypeRange() files: A Include/internal/pycore_pymath.h A Misc/NEWS.d/next/C API/2021-10-08-15-54-07.bpo-45412.KHyJCT.rst M Doc/whatsnew/3.11.rst M Include/pymath.h M Include/pyport.h M Objects/complexobject.c M Objects/floatobject.c M Python/pytime.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index e3650a6da8b95..2262d42a99add 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -577,3 +577,13 @@ Removed Use the new :c:type:`PyConfig` API of the :ref:`Python Initialization Configuration ` instead (:pep:`587`). (Contributed by Victor Stinner in :issue:`44113`.) + +* Remove the following math macros using the ``errno`` variable: + + * ``Py_ADJUST_ERANGE1()`` + * ``Py_ADJUST_ERANGE2()`` + * ``Py_OVERFLOWED()`` + * ``Py_SET_ERANGE_IF_OVERFLOW()`` + * ``Py_SET_ERRNO_ON_MATH_ERROR()`` + + (Contributed by Victor Stinner in :issue:`45412`.) diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h new file mode 100644 index 0000000000000..c299c64280183 --- /dev/null +++ b/Include/internal/pycore_pymath.h @@ -0,0 +1,73 @@ +#ifndef Py_INTERNAL_PYMATH_H +#define Py_INTERNAL_PYMATH_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +/* _Py_ADJUST_ERANGE1(x) + * _Py_ADJUST_ERANGE2(x, y) + * Set errno to 0 before calling a libm function, and invoke one of these + * macros after, passing the function result(s) (_Py_ADJUST_ERANGE2 is useful + * for functions returning complex results). This makes two kinds of + * adjustments to errno: (A) If it looks like the platform libm set + * errno=ERANGE due to underflow, clear errno. (B) If it looks like the + * platform libm overflowed but didn't set errno, force errno to ERANGE. In + * effect, we're trying to force a useful implementation of C89 errno + * behavior. + * Caution: + * This isn't reliable. See Py_OVERFLOWED comments. + * X and Y may be evaluated more than once. + */ +static inline void _Py_ADJUST_ERANGE1(double x) +{ + if (errno == 0) { + if (x == Py_HUGE_VAL || x == -Py_HUGE_VAL) { + errno = ERANGE; + } + } + else if (errno == ERANGE && x == 0.0) { + errno = 0; + } +} + +static inline void _Py_ADJUST_ERANGE2(double x, double y) +{ + if (x == Py_HUGE_VAL || x == -Py_HUGE_VAL || + y == Py_HUGE_VAL || y == -Py_HUGE_VAL) + { + if (errno == 0) { + errno = ERANGE; + } + } + else if (errno == ERANGE) { + errno = 0; + } +} + +// Return whether integral type *type* is signed or not. +#define _Py_IntegralTypeSigned(type) \ + ((type)(-1) < 0) + +// Return the maximum value of integral type *type*. +#define _Py_IntegralTypeMax(type) \ + ((_Py_IntegralTypeSigned(type)) ? (((((type)1 << (sizeof(type)*CHAR_BIT - 2)) - 1) << 1) + 1) : ~(type)0) + +// Return the minimum value of integral type *type*. +#define _Py_IntegralTypeMin(type) \ + ((_Py_IntegralTypeSigned(type)) ? -_Py_IntegralTypeMax(type) - 1 : 0) + +// Check whether *v* is in the range of integral type *type*. This is most +// useful if *v* is floating-point, since demoting a floating-point *v* to an +// integral type that cannot represent *v*'s integral part is undefined +// behavior. +#define _Py_InIntegralTypeRange(type, v) \ + (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type)) + +#ifdef __cplusplus +} +#endif +#endif /* !Py_INTERNAL_PYMATH_H */ diff --git a/Include/pymath.h b/Include/pymath.h index ebb3b05f1b53c..39a0bdad98b9e 100644 --- a/Include/pymath.h +++ b/Include/pymath.h @@ -176,50 +176,4 @@ PyAPI_FUNC(void) _Py_set_387controlword(unsigned short); #endif /* __INTEL_COMPILER */ #endif -/* Py_OVERFLOWED(X) - * Return 1 iff a libm function overflowed. Set errno to 0 before calling - * a libm function, and invoke this macro after, passing the function - * result. - * Caution: - * This isn't reliable. C99 no longer requires libm to set errno under - * any exceptional condition, but does require +- HUGE_VAL return - * values on overflow. A 754 box *probably* maps HUGE_VAL to a - * double infinity, and we're cool if that's so, unless the input - * was an infinity and an infinity is the expected result. A C89 - * system sets errno to ERANGE, so we check for that too. We're - * out of luck if a C99 754 box doesn't map HUGE_VAL to +Inf, or - * if the returned result is a NaN, or if a C89 box returns HUGE_VAL - * in non-overflow cases. - * X is evaluated more than once. - * Some platforms have better way to spell this, so expect some #ifdef'ery. - * - * OpenBSD uses 'isinf()' because a compiler bug on that platform causes - * the longer macro version to be mis-compiled. This isn't optimal, and - * should be removed once a newer compiler is available on that platform. - * The system that had the failure was running OpenBSD 3.2 on Intel, with - * gcc 2.95.3. - * - * According to Tim's checkin, the FreeBSD systems use isinf() to work - * around a FPE bug on that platform. - */ -#if defined(__FreeBSD__) || defined(__OpenBSD__) -#define Py_OVERFLOWED(X) isinf(X) -#else -#define Py_OVERFLOWED(X) ((X) != 0.0 && (errno == ERANGE || \ - (X) == Py_HUGE_VAL || \ - (X) == -Py_HUGE_VAL)) -#endif - -/* Return whether integral type *type* is signed or not. */ -#define _Py_IntegralTypeSigned(type) ((type)(-1) < 0) -/* Return the maximum value of integral type *type*. */ -#define _Py_IntegralTypeMax(type) ((_Py_IntegralTypeSigned(type)) ? (((((type)1 << (sizeof(type)*CHAR_BIT - 2)) - 1) << 1) + 1) : ~(type)0) -/* Return the minimum value of integral type *type*. */ -#define _Py_IntegralTypeMin(type) ((_Py_IntegralTypeSigned(type)) ? -_Py_IntegralTypeMax(type) - 1 : 0) -/* Check whether *v* is in the range of integral type *type*. This is most - * useful if *v* is floating-point, since demoting a floating-point *v* to an - * integral type that cannot represent *v*'s integral part is undefined - * behavior. */ -#define _Py_InIntegralTypeRange(type, v) (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type)) - #endif /* Py_PYMATH_H */ diff --git a/Include/pyport.h b/Include/pyport.h index d27d0838f1b2c..ffc9ba55618d8 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -316,69 +316,6 @@ extern "C" { #define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) (NARROW)(VALUE) #endif -/* Py_SET_ERRNO_ON_MATH_ERROR(x) - * If a libm function did not set errno, but it looks like the result - * overflowed or not-a-number, set errno to ERANGE or EDOM. Set errno - * to 0 before calling a libm function, and invoke this macro after, - * passing the function result. - * Caution: - * This isn't reliable. See Py_OVERFLOWED comments. - * X is evaluated more than once. - */ -#if defined(__FreeBSD__) || defined(__OpenBSD__) || (defined(__hpux) && defined(__ia64)) -#define _Py_SET_EDOM_FOR_NAN(X) if (isnan(X)) errno = EDOM; -#else -#define _Py_SET_EDOM_FOR_NAN(X) ; -#endif -#define Py_SET_ERRNO_ON_MATH_ERROR(X) \ - do { \ - if (errno == 0) { \ - if ((X) == Py_HUGE_VAL || (X) == -Py_HUGE_VAL) \ - errno = ERANGE; \ - else _Py_SET_EDOM_FOR_NAN(X) \ - } \ - } while(0) - -/* Py_SET_ERANGE_IF_OVERFLOW(x) - * An alias of Py_SET_ERRNO_ON_MATH_ERROR for backward-compatibility. - */ -#define Py_SET_ERANGE_IF_OVERFLOW(X) Py_SET_ERRNO_ON_MATH_ERROR(X) - -/* Py_ADJUST_ERANGE1(x) - * Py_ADJUST_ERANGE2(x, y) - * Set errno to 0 before calling a libm function, and invoke one of these - * macros after, passing the function result(s) (Py_ADJUST_ERANGE2 is useful - * for functions returning complex results). This makes two kinds of - * adjustments to errno: (A) If it looks like the platform libm set - * errno=ERANGE due to underflow, clear errno. (B) If it looks like the - * platform libm overflowed but didn't set errno, force errno to ERANGE. In - * effect, we're trying to force a useful implementation of C89 errno - * behavior. - * Caution: - * This isn't reliable. See Py_OVERFLOWED comments. - * X and Y may be evaluated more than once. - */ -#define Py_ADJUST_ERANGE1(X) \ - do { \ - if (errno == 0) { \ - if ((X) == Py_HUGE_VAL || (X) == -Py_HUGE_VAL) \ - errno = ERANGE; \ - } \ - else if (errno == ERANGE && (X) == 0.0) \ - errno = 0; \ - } while(0) - -#define Py_ADJUST_ERANGE2(X, Y) \ - do { \ - if ((X) == Py_HUGE_VAL || (X) == -Py_HUGE_VAL || \ - (Y) == Py_HUGE_VAL || (Y) == -Py_HUGE_VAL) { \ - if (errno == 0) \ - errno = ERANGE; \ - } \ - else if (errno == ERANGE) \ - errno = 0; \ - } while(0) - /* The functions _Py_dg_strtod and _Py_dg_dtoa in Python/dtoa.c (which are * required to support the short float repr introduced in Python 3.1) require * that the floating-point unit that's being used for arithmetic operations diff --git a/Misc/NEWS.d/next/C API/2021-10-08-15-54-07.bpo-45412.KHyJCT.rst b/Misc/NEWS.d/next/C API/2021-10-08-15-54-07.bpo-45412.KHyJCT.rst new file mode 100644 index 0000000000000..49746810ee5b9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-08-15-54-07.bpo-45412.KHyJCT.rst @@ -0,0 +1,9 @@ +Remove the following math macros using the ``errno`` variable: + +* ``Py_ADJUST_ERANGE1()`` +* ``Py_ADJUST_ERANGE2()`` +* ``Py_OVERFLOWED()`` +* ``Py_SET_ERANGE_IF_OVERFLOW()`` +* ``Py_SET_ERRNO_ON_MATH_ERROR()`` + +Patch by Victor Stinner. diff --git a/Objects/complexobject.c b/Objects/complexobject.c index cfe6c737578a0..f08f03fcb49a5 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -8,6 +8,7 @@ #include "Python.h" #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_Init() +#include "pycore_pymath.h" // _Py_ADJUST_ERANGE2() #include "structmember.h" // PyMemberDef @@ -525,7 +526,7 @@ complex_pow(PyObject *v, PyObject *w, PyObject *z) p = _Py_c_pow(a, b); } - Py_ADJUST_ERANGE2(p.real, p.imag); + _Py_ADJUST_ERANGE2(p.real, p.imag); if (errno == EDOM) { PyErr_SetString(PyExc_ZeroDivisionError, "0.0 to a negative or complex power"); diff --git a/Objects/floatobject.c b/Objects/floatobject.c index e4ce7e74d2c57..d25d97f4cbf61 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -8,6 +8,7 @@ #include "pycore_interp.h" // _PyInterpreterState.float_state #include "pycore_long.h" // _PyLong_GetOne() #include "pycore_object.h" // _PyObject_Init() +#include "pycore_pymath.h" // _Py_ADJUST_ERANGE1() #include "pycore_pystate.h" // _PyInterpreterState_GET() #include @@ -809,7 +810,7 @@ float_pow(PyObject *v, PyObject *w, PyObject *z) */ errno = 0; ix = pow(iv, iw); - Py_ADJUST_ERANGE1(ix); + _Py_ADJUST_ERANGE1(ix); if (negate_result) ix = -ix; diff --git a/Python/pytime.c b/Python/pytime.c index 8865638e91c23..9653662b0fb14 100644 --- a/Python/pytime.c +++ b/Python/pytime.c @@ -1,10 +1,11 @@ #include "Python.h" +#include "pycore_pymath.h" // _Py_InIntegralTypeRange() #ifdef MS_WINDOWS -#include /* struct timeval */ +# include // struct timeval #endif #if defined(__APPLE__) -#include /* mach_absolute_time(), mach_timebase_info() */ +# include // mach_absolute_time(), mach_timebase_info() #if defined(__APPLE__) && defined(__has_builtin) # if __has_builtin(__builtin_available) From webhook-mailer at python.org Mon Oct 11 15:34:58 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 19:34:58 -0000 Subject: [Python-checkins] bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) Message-ID: https://github.com/python/cpython/commit/320084fe7de90319928d8f3e597d5bca04db13f3 commit: 320084fe7de90319928d8f3e597d5bca04db13f3 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T12:34:51-07:00 summary: bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) (cherry picked from commit 659812b451aefe1f0e5f83540296519a5fb8f313) Co-authored-by: Olaf van der Spek files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index 714de8d41a350..bf431cfb8a410 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -373,8 +373,8 @@ TCP echo server using the :func:`asyncio.start_server` function:: server = await asyncio.start_server( handle_echo, '127.0.0.1', 8888) - addr = server.sockets[0].getsockname() - print(f'Serving on {addr}') + addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets) + print(f'Serving on {addrs}') async with server: await server.serve_forever() From webhook-mailer at python.org Mon Oct 11 15:34:58 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 11 Oct 2021 19:34:58 -0000 Subject: [Python-checkins] bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) Message-ID: https://github.com/python/cpython/commit/bb4f885892be0c337db3a81ef2936be0b3855de3 commit: bb4f885892be0c337db3a81ef2936be0b3855de3 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-11T12:34:47-07:00 summary: bpo-45351, asyncio: Enhance echo server example, print all addresses (GH-28828) (cherry picked from commit 659812b451aefe1f0e5f83540296519a5fb8f313) Co-authored-by: Olaf van der Spek files: M Doc/library/asyncio-stream.rst diff --git a/Doc/library/asyncio-stream.rst b/Doc/library/asyncio-stream.rst index b3e229c24f07d..95a8e4649beed 100644 --- a/Doc/library/asyncio-stream.rst +++ b/Doc/library/asyncio-stream.rst @@ -395,8 +395,8 @@ TCP echo server using the :func:`asyncio.start_server` function:: server = await asyncio.start_server( handle_echo, '127.0.0.1', 8888) - addr = server.sockets[0].getsockname() - print(f'Serving on {addr}') + addrs = ', '.join(str(sock.getsockname()) for sock in server.sockets) + print(f'Serving on {addrs}') async with server: await server.serve_forever() From webhook-mailer at python.org Mon Oct 11 16:51:41 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 20:51:41 -0000 Subject: [Python-checkins] bpo-45434: Cleanup Python.h header file (GH-28883) Message-ID: https://github.com/python/cpython/commit/47717d1186563695e798b40350d15b00d04a5237 commit: 47717d1186563695e798b40350d15b00d04a5237 branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T22:51:32+02:00 summary: bpo-45434: Cleanup Python.h header file (GH-28883) * Move limits.h include and UCHAR_MAX checks to pyport.h. * Move sanitizers macros to pyport.h. * Remove comment about : C extensions are built with NDEBUG automatically by Python. files: M Include/Python.h M Include/pyport.h diff --git a/Include/Python.h b/Include/Python.h index a83befa311736..fa77521d46ebb 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -1,83 +1,50 @@ +// Entry point of the Python C API. +// C extensions should only #include , and not include directly +// the other Python header files included by . + #ifndef Py_PYTHON_H #define Py_PYTHON_H -/* Since this is a "meta-include" file, no #ifdef __cplusplus / extern "C" { */ -/* Include nearly all Python header files */ +// Since this is a "meta-include" file, no #ifdef __cplusplus / extern "C" { +// Include Python header files #include "patchlevel.h" #include "pyconfig.h" #include "pymacconfig.h" -#include - -#ifndef UCHAR_MAX -#error "Something's broken. UCHAR_MAX should be defined in limits.h." -#endif - -#if UCHAR_MAX != 255 -#error "Python's source code assumes C's unsigned char is an 8-bit type." -#endif - #if defined(__sgi) && !defined(_SGI_MP_SOURCE) -#define _SGI_MP_SOURCE +# define _SGI_MP_SOURCE #endif -#include +#include // NULL, FILE* #ifndef NULL # error "Python.h requires that stdio.h define NULL." #endif -#include +#include // memcpy() #ifdef HAVE_ERRNO_H -#include +# include // errno #endif #include #ifndef MS_WINDOWS -#include +# include #endif - -/* For size_t? */ #ifdef HAVE_STDDEF_H -#include + // For size_t +# include #endif -/* CAUTION: Build setups should ensure that NDEBUG is defined on the - * compiler command line when building Python in release mode; else - * assert() calls won't be removed. - */ #include #include "pyport.h" #include "pymacro.h" - -/* A convenient way for code to know if sanitizers are enabled. */ -#if defined(__has_feature) -# if __has_feature(memory_sanitizer) -# if !defined(_Py_MEMORY_SANITIZER) -# define _Py_MEMORY_SANITIZER -# endif -# endif -# if __has_feature(address_sanitizer) -# if !defined(_Py_ADDRESS_SANITIZER) -# define _Py_ADDRESS_SANITIZER -# endif -# endif -#elif defined(__GNUC__) -# if defined(__SANITIZE_ADDRESS__) -# define _Py_ADDRESS_SANITIZER -# endif -#endif - #include "pymath.h" #include "pymem.h" - #include "object.h" #include "objimpl.h" #include "typeslots.h" #include "pyhash.h" - #include "cpython/pydebug.h" - #include "bytearrayobject.h" #include "bytesobject.h" #include "unicodeobject.h" @@ -115,15 +82,12 @@ #include "namespaceobject.h" #include "cpython/picklebufobject.h" #include "cpython/pytime.h" - #include "codecs.h" #include "pyerrors.h" - #include "cpython/initconfig.h" #include "pythread.h" #include "pystate.h" #include "context.h" - #include "modsupport.h" #include "compile.h" #include "pythonrun.h" @@ -133,12 +97,9 @@ #include "osmodule.h" #include "intrcheck.h" #include "import.h" - #include "abstract.h" #include "bltinmodule.h" - #include "eval.h" - #include "cpython/pyctype.h" #include "pystrtod.h" #include "pystrcmp.h" diff --git a/Include/pyport.h b/Include/pyport.h index ffc9ba55618d8..a38074cc503e1 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -5,6 +5,14 @@ #include +#include +#ifndef UCHAR_MAX +# error "limits.h must define UCHAR_MAX" +#endif +#if UCHAR_MAX != 255 +# error "Python's source code assumes C's unsigned char is an 8-bit type" +#endif + /* Defines to build Python and its standard library: * @@ -851,4 +859,22 @@ extern _invalid_parameter_handler _Py_silent_invalid_parameter_handler; #endif +/* A convenient way for code to know if sanitizers are enabled. */ +#if defined(__has_feature) +# if __has_feature(memory_sanitizer) +# if !defined(_Py_MEMORY_SANITIZER) +# define _Py_MEMORY_SANITIZER +# endif +# endif +# if __has_feature(address_sanitizer) +# if !defined(_Py_ADDRESS_SANITIZER) +# define _Py_ADDRESS_SANITIZER +# endif +# endif +#elif defined(__GNUC__) +# if defined(__SANITIZE_ADDRESS__) +# define _Py_ADDRESS_SANITIZER +# endif +#endif + #endif /* Py_PYPORT_H */ From webhook-mailer at python.org Mon Oct 11 17:07:29 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 21:07:29 -0000 Subject: [Python-checkins] bpo-45410: Add test.support.flush_std_streams() (GH-28885) Message-ID: https://github.com/python/cpython/commit/1ebd798fddef51e1f6fd40a4860278a1851ac268 commit: 1ebd798fddef51e1f6fd40a4860278a1851ac268 branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T23:07:21+02:00 summary: bpo-45410: Add test.support.flush_std_streams() (GH-28885) support.print_warning() now flushs sys.stdout. files: M Doc/library/test.rst M Lib/test/libregrtest/utils.py M Lib/test/support/__init__.py diff --git a/Doc/library/test.rst b/Doc/library/test.rst index a8dc35476fc9b..699db14596f25 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -607,6 +607,15 @@ The :mod:`test.support` module defines the following functions: target of the "as" clause, if there is one. +.. function:: flush_std_streams() + + Call the ``flush()`` method on :data:`sys.stdout` and then on + :data:`sys.stderr`. It can be used to make sure that the logs order is + consistent before writing into stderr. + + .. versionadded:: 3.11 + + .. function:: print_warning(msg) Print a warning into :data:`sys.__stderr__`. Format the message as: diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index ad18b50f5db99..c71467a51921d 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -68,20 +68,13 @@ def print_warning(msg): orig_unraisablehook = None -def flush_std_streams(): - if sys.stdout is not None: - sys.stdout.flush() - if sys.stderr is not None: - sys.stderr.flush() - - def regrtest_unraisable_hook(unraisable): global orig_unraisablehook support.environment_altered = True print_warning("Unraisable exception") old_stderr = sys.stderr try: - flush_std_streams() + support.flush_std_streams() sys.stderr = sys.__stderr__ orig_unraisablehook(unraisable) sys.stderr.flush() @@ -104,7 +97,7 @@ def regrtest_threading_excepthook(args): print_warning(f"Uncaught thread exception: {args.exc_type.__name__}") old_stderr = sys.stderr try: - flush_std_streams() + support.flush_std_streams() sys.stderr = sys.__stderr__ orig_threading_excepthook(args) sys.stderr.flush() diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index b29f438e52cf8..0bbe813775ce5 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1167,7 +1167,16 @@ def run_doctest(module, verbosity=None, optionflags=0): #======================================================================= # Support for saving and restoring the imported modules. +def flush_std_streams(): + if sys.stdout is not None: + sys.stdout.flush() + if sys.stderr is not None: + sys.stderr.flush() + + def print_warning(msg): + # bpo-45410: Explicitly flush stdout to keep logs in order + flush_std_streams() # bpo-39983: Print into sys.__stderr__ to display the warning even # when sys.stderr is captured temporarily by a test for line in msg.splitlines(): From webhook-mailer at python.org Mon Oct 11 17:07:45 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 21:07:45 -0000 Subject: [Python-checkins] bpo-45412: Update _Py_ADJUST_ERANGE1() comment (GH-28884) Message-ID: https://github.com/python/cpython/commit/a9fe1a8e5b4698937e06c2c419da92e6f78f2ee7 commit: a9fe1a8e5b4698937e06c2c419da92e6f78f2ee7 branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T23:07:41+02:00 summary: bpo-45412: Update _Py_ADJUST_ERANGE1() comment (GH-28884) Copy the comment from the removed Py_OVERFLOWED() function. files: M Include/internal/pycore_pymath.h diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h index c299c64280183..e4d5778cfbf7c 100644 --- a/Include/internal/pycore_pymath.h +++ b/Include/internal/pycore_pymath.h @@ -19,8 +19,15 @@ extern "C" { * effect, we're trying to force a useful implementation of C89 errno * behavior. * Caution: - * This isn't reliable. See Py_OVERFLOWED comments. - * X and Y may be evaluated more than once. + * This isn't reliable. C99 no longer requires libm to set errno under + * any exceptional condition, but does require +- HUGE_VAL return + * values on overflow. A 754 box *probably* maps HUGE_VAL to a + * double infinity, and we're cool if that's so, unless the input + * was an infinity and an infinity is the expected result. A C89 + * system sets errno to ERANGE, so we check for that too. We're + * out of luck if a C99 754 box doesn't map HUGE_VAL to +Inf, or + * if the returned result is a NaN, or if a C89 box returns HUGE_VAL + * in non-overflow cases. */ static inline void _Py_ADJUST_ERANGE1(double x) { From webhook-mailer at python.org Mon Oct 11 17:09:45 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 21:09:45 -0000 Subject: [Python-checkins] bpo-45412: Move _Py_SET_53BIT_PRECISION_START to pycore_pymath.h (GH-28882) Message-ID: https://github.com/python/cpython/commit/7103356455c8b0c2ba3523929327756413337a31 commit: 7103356455c8b0c2ba3523929327756413337a31 branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T23:09:40+02:00 summary: bpo-45412: Move _Py_SET_53BIT_PRECISION_START to pycore_pymath.h (GH-28882) Move the following macros , to pycore_pymath.h (internal C API): * _Py_SET_53BIT_PRECISION_HEADER * _Py_SET_53BIT_PRECISION_START * _Py_SET_53BIT_PRECISION_END PEP 7: add braces to if and "do { ... } while (0)" in these macros. Move also _Py_get_387controlword() and _Py_set_387controlword() definitions to pycore_pymath.h. These functions are no longer exported. pystrtod.c now includes pycore_pymath.h. files: M Include/internal/pycore_pymath.h M Include/pymath.h M Include/pyport.h M Python/pymath.c M Python/pystrtod.c diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h index e4d5778cfbf7c..b1a200459fecd 100644 --- a/Include/internal/pycore_pymath.h +++ b/Include/internal/pycore_pymath.h @@ -74,6 +74,97 @@ static inline void _Py_ADJUST_ERANGE2(double x, double y) #define _Py_InIntegralTypeRange(type, v) \ (_Py_IntegralTypeMin(type) <= v && v <= _Py_IntegralTypeMax(type)) + +//--- Implementation of the HAVE_PY_SET_53BIT_PRECISION macro ------------- +//--- defined in pyport.h ------------------------------------------------- +// +// Give appropriate definitions for the following three macros: +// +// _Py_SET_53BIT_PRECISION_HEADER : any variable declarations needed to +// use the two macros below. +// _Py_SET_53BIT_PRECISION_START : store original FPU settings, and +// set FPU to 53-bit precision/round-half-to-even +// _Py_SET_53BIT_PRECISION_END : restore original FPU settings + +// Get and set x87 control word for gcc/x86 +#ifdef HAVE_GCC_ASM_FOR_X87 + +// Functions defined in Python/pymath.c +extern unsigned short _Py_get_387controlword(void); +extern void _Py_set_387controlword(unsigned short); + +#define _Py_SET_53BIT_PRECISION_HEADER \ + unsigned short old_387controlword, new_387controlword +#define _Py_SET_53BIT_PRECISION_START \ + do { \ + old_387controlword = _Py_get_387controlword(); \ + new_387controlword = (old_387controlword & ~0x0f00) | 0x0200; \ + if (new_387controlword != old_387controlword) { \ + _Py_set_387controlword(new_387controlword); \ + } \ + } while (0) +#define _Py_SET_53BIT_PRECISION_END \ + do { \ + if (new_387controlword != old_387controlword) { \ + _Py_set_387controlword(old_387controlword); \ + } \ + } while (0) +#endif + +// Get and set x87 control word for VisualStudio/x86. +// x87 is not supported in 64-bit or ARM. +#if defined(_MSC_VER) && !defined(_WIN64) && !defined(_M_ARM) +#define _Py_SET_53BIT_PRECISION_HEADER \ + unsigned int old_387controlword, new_387controlword, out_387controlword + // We use the __control87_2 function to set only the x87 control word. + // The SSE control word is unaffected. +#define _Py_SET_53BIT_PRECISION_START \ + do { \ + __control87_2(0, 0, &old_387controlword, NULL); \ + new_387controlword = \ + (old_387controlword & ~(_MCW_PC | _MCW_RC)) | (_PC_53 | _RC_NEAR); \ + if (new_387controlword != old_387controlword) { \ + __control87_2(new_387controlword, _MCW_PC | _MCW_RC, \ + &out_387controlword, NULL); \ + } \ + } while (0) +#define _Py_SET_53BIT_PRECISION_END \ + do { \ + if (new_387controlword != old_387controlword) { \ + __control87_2(old_387controlword, _MCW_PC | _MCW_RC, \ + &out_387controlword, NULL); \ + } \ + } while (0) +#endif + +#ifdef HAVE_GCC_ASM_FOR_MC68881 +#define _Py_SET_53BIT_PRECISION_HEADER \ + unsigned int old_fpcr, new_fpcr +#define _Py_SET_53BIT_PRECISION_START \ + do { \ + __asm__ ("fmove.l %%fpcr,%0" : "=g" (old_fpcr)); \ + /* Set double precision / round to nearest. */ \ + new_fpcr = (old_fpcr & ~0xf0) | 0x80; \ + if (new_fpcr != old_fpcr) { \ + __asm__ volatile ("fmove.l %0,%%fpcr" : : "g" (new_fpcr));\ + } \ + } while (0) +#define _Py_SET_53BIT_PRECISION_END \ + do { \ + if (new_fpcr != old_fpcr) { \ + __asm__ volatile ("fmove.l %0,%%fpcr" : : "g" (old_fpcr)); \ + } \ + } while (0) +#endif + +// Default definitions are empty +#ifndef _Py_SET_53BIT_PRECISION_HEADER +# define _Py_SET_53BIT_PRECISION_HEADER +# define _Py_SET_53BIT_PRECISION_START +# define _Py_SET_53BIT_PRECISION_END +#endif + + #ifdef __cplusplus } #endif diff --git a/Include/pymath.h b/Include/pymath.h index 39a0bdad98b9e..2f47f87434f61 100644 --- a/Include/pymath.h +++ b/Include/pymath.h @@ -78,13 +78,6 @@ PyAPI_FUNC(double) _Py_force_double(double); #endif #endif -#ifndef Py_LIMITED_API -#ifdef HAVE_GCC_ASM_FOR_X87 -PyAPI_FUNC(unsigned short) _Py_get_387controlword(void); -PyAPI_FUNC(void) _Py_set_387controlword(unsigned short); -#endif -#endif - /* Py_IS_NAN(X) * Return 1 if float or double arg is a NaN, else 0. * Caution: @@ -95,11 +88,11 @@ PyAPI_FUNC(void) _Py_set_387controlword(unsigned short); * Note: PC/pyconfig.h defines Py_IS_NAN as _isnan */ #ifndef Py_IS_NAN -#if defined HAVE_DECL_ISNAN && HAVE_DECL_ISNAN == 1 -#define Py_IS_NAN(X) isnan(X) -#else -#define Py_IS_NAN(X) ((X) != (X)) -#endif +# if defined HAVE_DECL_ISNAN && HAVE_DECL_ISNAN == 1 +# define Py_IS_NAN(X) isnan(X) +# else +# define Py_IS_NAN(X) ((X) != (X)) +# endif #endif /* Py_IS_INFINITY(X) diff --git a/Include/pyport.h b/Include/pyport.h index a38074cc503e1..a8a2d6d0d9d92 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -335,86 +335,25 @@ extern "C" { * * #define HAVE_PY_SET_53BIT_PRECISION 1 * - * and also give appropriate definitions for the following three macros: - * - * _PY_SET_53BIT_PRECISION_START : store original FPU settings, and - * set FPU to 53-bit precision/round-half-to-even - * _PY_SET_53BIT_PRECISION_END : restore original FPU settings - * _PY_SET_53BIT_PRECISION_HEADER : any variable declarations needed to - * use the two macros above. - * * The macros are designed to be used within a single C function: see * Python/pystrtod.c for an example of their use. */ -/* get and set x87 control word for gcc/x86 */ +// HAVE_PY_SET_53BIT_PRECISION macro must be kept in sync with pycore_pymath.h #ifdef HAVE_GCC_ASM_FOR_X87 -#define HAVE_PY_SET_53BIT_PRECISION 1 -/* _Py_get/set_387controlword functions are defined in Python/pymath.c */ -#define _Py_SET_53BIT_PRECISION_HEADER \ - unsigned short old_387controlword, new_387controlword -#define _Py_SET_53BIT_PRECISION_START \ - do { \ - old_387controlword = _Py_get_387controlword(); \ - new_387controlword = (old_387controlword & ~0x0f00) | 0x0200; \ - if (new_387controlword != old_387controlword) \ - _Py_set_387controlword(new_387controlword); \ - } while (0) -#define _Py_SET_53BIT_PRECISION_END \ - if (new_387controlword != old_387controlword) \ - _Py_set_387controlword(old_387controlword) -#endif - -/* get and set x87 control word for VisualStudio/x86 */ -#if defined(_MSC_VER) && !defined(_WIN64) && !defined(_M_ARM) /* x87 not supported in 64-bit or ARM */ -#define HAVE_PY_SET_53BIT_PRECISION 1 -#define _Py_SET_53BIT_PRECISION_HEADER \ - unsigned int old_387controlword, new_387controlword, out_387controlword -/* We use the __control87_2 function to set only the x87 control word. - The SSE control word is unaffected. */ -#define _Py_SET_53BIT_PRECISION_START \ - do { \ - __control87_2(0, 0, &old_387controlword, NULL); \ - new_387controlword = \ - (old_387controlword & ~(_MCW_PC | _MCW_RC)) | (_PC_53 | _RC_NEAR); \ - if (new_387controlword != old_387controlword) \ - __control87_2(new_387controlword, _MCW_PC | _MCW_RC, \ - &out_387controlword, NULL); \ - } while (0) -#define _Py_SET_53BIT_PRECISION_END \ - do { \ - if (new_387controlword != old_387controlword) \ - __control87_2(old_387controlword, _MCW_PC | _MCW_RC, \ - &out_387controlword, NULL); \ - } while (0) + // Get and set x87 control word for gcc/x86 +# define HAVE_PY_SET_53BIT_PRECISION 1 +#endif +#if defined(_MSC_VER) && !defined(_WIN64) && !defined(_M_ARM) + // Get and set x87 control word for VisualStudio/x86. + // x87 not supported in 64-bit or ARM. +# define HAVE_PY_SET_53BIT_PRECISION 1 #endif - #ifdef HAVE_GCC_ASM_FOR_MC68881 -#define HAVE_PY_SET_53BIT_PRECISION 1 -#define _Py_SET_53BIT_PRECISION_HEADER \ - unsigned int old_fpcr, new_fpcr -#define _Py_SET_53BIT_PRECISION_START \ - do { \ - __asm__ ("fmove.l %%fpcr,%0" : "=g" (old_fpcr)); \ - /* Set double precision / round to nearest. */ \ - new_fpcr = (old_fpcr & ~0xf0) | 0x80; \ - if (new_fpcr != old_fpcr) \ - __asm__ volatile ("fmove.l %0,%%fpcr" : : "g" (new_fpcr)); \ - } while (0) -#define _Py_SET_53BIT_PRECISION_END \ - do { \ - if (new_fpcr != old_fpcr) \ - __asm__ volatile ("fmove.l %0,%%fpcr" : : "g" (old_fpcr)); \ - } while (0) -#endif - -/* default definitions are empty */ -#ifndef HAVE_PY_SET_53BIT_PRECISION -#define _Py_SET_53BIT_PRECISION_HEADER -#define _Py_SET_53BIT_PRECISION_START -#define _Py_SET_53BIT_PRECISION_END +# define HAVE_PY_SET_53BIT_PRECISION 1 #endif + /* If we can't guarantee 53-bit precision, don't use the code in Python/dtoa.c, but fall back to standard code. This means that repr of a float will be long (17 sig digits). @@ -430,14 +369,14 @@ extern "C" { #if !defined(DOUBLE_IS_LITTLE_ENDIAN_IEEE754) && \ !defined(DOUBLE_IS_BIG_ENDIAN_IEEE754) && \ !defined(DOUBLE_IS_ARM_MIXED_ENDIAN_IEEE754) -#define PY_NO_SHORT_FLOAT_REPR +# define PY_NO_SHORT_FLOAT_REPR #endif /* double rounding is symptomatic of use of extended precision on x86. If we're seeing double rounding, and we don't have any mechanism available for changing the FPU rounding precision, then don't use Python/dtoa.c. */ #if defined(X87_DOUBLE_ROUNDING) && !defined(HAVE_PY_SET_53BIT_PRECISION) -#define PY_NO_SHORT_FLOAT_REPR +# define PY_NO_SHORT_FLOAT_REPR #endif diff --git a/Python/pymath.c b/Python/pymath.c index 24b804223eef1..d3c52a09650c5 100644 --- a/Python/pymath.c +++ b/Python/pymath.c @@ -13,10 +13,10 @@ double _Py_force_double(double x) } #endif -#ifdef HAVE_GCC_ASM_FOR_X87 -/* inline assembly for getting and setting the 387 FPU control word on - gcc/x86 */ +#ifdef HAVE_GCC_ASM_FOR_X87 +// Inline assembly for getting and setting the 387 FPU control word on +// GCC/x86. #ifdef _Py_MEMORY_SANITIZER __attribute__((no_sanitize_memory)) #endif @@ -29,8 +29,7 @@ unsigned short _Py_get_387controlword(void) { void _Py_set_387controlword(unsigned short cw) { __asm__ __volatile__ ("fldcw %0" : : "m" (cw)); } - -#endif +#endif // HAVE_GCC_ASM_FOR_X87 #ifndef HAVE_HYPOT diff --git a/Python/pystrtod.c b/Python/pystrtod.c index 9145d4eba121e..ab5814de21ba5 100644 --- a/Python/pystrtod.c +++ b/Python/pystrtod.c @@ -2,6 +2,7 @@ #include #include "pycore_dtoa.h" +#include "pycore_pymath.h" // _Py_SET_53BIT_PRECISION_START #include /* Case-insensitive string match used for nan and inf detection; t should be From webhook-mailer at python.org Mon Oct 11 17:30:10 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 21:30:10 -0000 Subject: [Python-checkins] bpo-45434: Python.h no longer includes (GH-28888) Message-ID: https://github.com/python/cpython/commit/03ea862b8a8234176761240ba122254e9eb11663 commit: 03ea862b8a8234176761240ba122254e9eb11663 branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T23:30:00+02:00 summary: bpo-45434: Python.h no longer includes (GH-28888) files: A Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst M Doc/whatsnew/3.11.rst M Include/Python.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 2262d42a99add..1455a598e1b3c 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -552,6 +552,10 @@ Porting to Python 3.11 (Contributed by Victor Stinner in :issue:`39573`.) +* The ```` header file no longer includes ````. C + extensions using ```` must now include it explicitly. + (Contributed by Victor Stinner in :issue:`45434`.) + Deprecated ---------- diff --git a/Include/Python.h b/Include/Python.h index fa77521d46ebb..8c84c90f9a57e 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -25,7 +25,6 @@ #ifdef HAVE_ERRNO_H # include // errno #endif -#include #ifndef MS_WINDOWS # include #endif diff --git a/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst new file mode 100644 index 0000000000000..c04dda532557d --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst @@ -0,0 +1,3 @@ +The ```` header file no longer includes ````. C +extensions using ```` must now include it explicitly. Patch by +Victor Stinner. From webhook-mailer at python.org Mon Oct 11 17:36:45 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 21:36:45 -0000 Subject: [Python-checkins] bpo-41123: Remove Py_UNICODE_COPY() and Py_UNICODE_FILL() (GH-28887) Message-ID: https://github.com/python/cpython/commit/1f316ea3b4fa319eec4f375fb683467b424c964e commit: 1f316ea3b4fa319eec4f375fb683467b424c964e branch: main author: Victor Stinner committer: vstinner date: 2021-10-11T23:36:37+02:00 summary: bpo-41123: Remove Py_UNICODE_COPY() and Py_UNICODE_FILL() (GH-28887) files: A Misc/NEWS.d/next/C API/2021-10-11-22-58-33.bpo-41123.myrlIp.rst M Doc/whatsnew/3.11.rst M Include/cpython/unicodeobject.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 1455a598e1b3c..0d30fe8b64644 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -591,3 +591,8 @@ Removed * ``Py_SET_ERRNO_ON_MATH_ERROR()`` (Contributed by Victor Stinner in :issue:`45412`.) + +* Remove ``Py_UNICODE_COPY()`` and ``Py_UNICODE_FILL()`` macros, deprecated + since Python 3.3. Use ``PyUnicode_CopyCharacters()`` or ``memcpy()`` + (``wchar_t*`` string), and ``PyUnicode_Fill()`` functions instead. + (Contributed by Victor Stinner in :issue:`41123`.) diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index b40e2ea1011f5..0cbdbdbbe0af2 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -50,19 +50,6 @@ Py_UNICODE_ISDIGIT(ch) || \ Py_UNICODE_ISNUMERIC(ch)) -Py_DEPRECATED(3.3) static inline void -Py_UNICODE_COPY(Py_UNICODE *target, const Py_UNICODE *source, Py_ssize_t length) { - memcpy(target, source, (size_t)(length) * sizeof(Py_UNICODE)); -} - -Py_DEPRECATED(3.3) static inline void -Py_UNICODE_FILL(Py_UNICODE *target, Py_UNICODE value, Py_ssize_t length) { - Py_ssize_t i; - for (i = 0; i < length; i++) { - target[i] = value; - } -} - /* macros to work with surrogates */ #define Py_UNICODE_IS_SURROGATE(ch) (0xD800 <= (ch) && (ch) <= 0xDFFF) #define Py_UNICODE_IS_HIGH_SURROGATE(ch) (0xD800 <= (ch) && (ch) <= 0xDBFF) diff --git a/Misc/NEWS.d/next/C API/2021-10-11-22-58-33.bpo-41123.myrlIp.rst b/Misc/NEWS.d/next/C API/2021-10-11-22-58-33.bpo-41123.myrlIp.rst new file mode 100644 index 0000000000000..3ae302bcea245 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-11-22-58-33.bpo-41123.myrlIp.rst @@ -0,0 +1,4 @@ +Remove ``Py_UNICODE_COPY()`` and ``Py_UNICODE_FILL()`` macros, deprecated +since Python 3.3. Use ``PyUnicode_CopyCharacters()`` or ``memcpy()`` +(``wchar_t*`` string), and ``PyUnicode_Fill()`` functions instead. Patch by +Victor Stinner. From webhook-mailer at python.org Mon Oct 11 18:12:09 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 22:12:09 -0000 Subject: [Python-checkins] bpo-45412: Move copysign() define to pycore_pymath.h (GH-28889) Message-ID: https://github.com/python/cpython/commit/61190e092b8258ede92ac543bb39bad0f7168104 commit: 61190e092b8258ede92ac543bb39bad0f7168104 branch: main author: Victor Stinner committer: vstinner date: 2021-10-12T00:12:00+02:00 summary: bpo-45412: Move copysign() define to pycore_pymath.h (GH-28889) Move definitions of copysign(), round(), hypot(), fmod(), etc. from pymath.h to pycore_pymath.h. These functions are not exported by libpython and so must not be part of the C API. files: M Include/internal/pycore_pymath.h M Include/pymath.h diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h index b1a200459fecd..38f76d0461a89 100644 --- a/Include/internal/pycore_pymath.h +++ b/Include/internal/pycore_pymath.h @@ -8,6 +8,34 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif + +// Python provides implementations for copysign(), round() and hypot() in +// Python/pymath.c just in case your math library doesn't provide the +// functions. +// +// Note: PC/pyconfig.h defines copysign as _copysign +#ifndef HAVE_COPYSIGN +extern double copysign(double, double); +#endif + +#ifndef HAVE_ROUND +extern double round(double); +#endif + +#ifndef HAVE_HYPOT +extern double hypot(double, double); +#endif + +// Extra declarations +#if !defined(_MSC_VER) && !defined(__STDC__) +extern double fmod (double, double); +extern double frexp (double, int *); +extern double ldexp (double, int); +extern double modf (double, double *); +extern double pow(double, double); +#endif // !defined(_MSC_VER) && !defined(__STDC__) + + /* _Py_ADJUST_ERANGE1(x) * _Py_ADJUST_ERANGE2(x, y) * Set errno to 0 before calling a libm function, and invoke one of these diff --git a/Include/pymath.h b/Include/pymath.h index 2f47f87434f61..d688e5033e0cc 100644 --- a/Include/pymath.h +++ b/Include/pymath.h @@ -1,41 +1,10 @@ +// Symbols and macros to supply platform-independent interfaces to mathematical +// functions and constants. + #ifndef Py_PYMATH_H #define Py_PYMATH_H -#include "pyconfig.h" /* include for defines */ - -/************************************************************************** -Symbols and macros to supply platform-independent interfaces to mathematical -functions and constants -**************************************************************************/ - -/* Python provides implementations for copysign, round and hypot in - * Python/pymath.c just in case your math library doesn't provide the - * functions. - * - *Note: PC/pyconfig.h defines copysign as _copysign - */ -#ifndef HAVE_COPYSIGN -extern double copysign(double, double); -#endif - -#ifndef HAVE_ROUND -extern double round(double); -#endif - -#ifndef HAVE_HYPOT -extern double hypot(double, double); -#endif - -/* extra declarations */ -#ifndef _MSC_VER -#ifndef __STDC__ -extern double fmod (double, double); -extern double frexp (double, int *); -extern double ldexp (double, int); -extern double modf (double, double *); -extern double pow(double, double); -#endif /* __STDC__ */ -#endif /* _MSC_VER */ +#include "pyconfig.h" // HAVE_DECL_ISNAN /* High precision definition of pi and e (Euler) * The values are taken from libc6's math.h. @@ -123,13 +92,13 @@ PyAPI_FUNC(double) _Py_force_double(double); * Note: PC/pyconfig.h defines Py_IS_FINITE as _finite */ #ifndef Py_IS_FINITE -#if defined HAVE_DECL_ISFINITE && HAVE_DECL_ISFINITE == 1 -#define Py_IS_FINITE(X) isfinite(X) -#elif defined HAVE_FINITE -#define Py_IS_FINITE(X) finite(X) -#else -#define Py_IS_FINITE(X) (!Py_IS_INFINITY(X) && !Py_IS_NAN(X)) -#endif +# if defined HAVE_DECL_ISFINITE && HAVE_DECL_ISFINITE == 1 +# define Py_IS_FINITE(X) isfinite(X) +# elif defined HAVE_FINITE +# define Py_IS_FINITE(X) finite(X) +# else +# define Py_IS_FINITE(X) (!Py_IS_INFINITY(X) && !Py_IS_NAN(X)) +# endif #endif /* HUGE_VAL is supposed to expand to a positive double infinity. Python @@ -140,7 +109,7 @@ PyAPI_FUNC(double) _Py_force_double(double); * config to #define Py_HUGE_VAL to something that works on your platform. */ #ifndef Py_HUGE_VAL -#define Py_HUGE_VAL HUGE_VAL +# define Py_HUGE_VAL HUGE_VAL #endif /* Py_NAN @@ -149,10 +118,10 @@ PyAPI_FUNC(double) _Py_force_double(double); * doesn't support NaNs. */ #if !defined(Py_NAN) && !defined(Py_NO_NAN) -#if !defined(__INTEL_COMPILER) - #define Py_NAN (Py_HUGE_VAL * 0.) -#else /* __INTEL_COMPILER */ - #if defined(ICC_NAN_STRICT) +# if !defined(__INTEL_COMPILER) +# define Py_NAN (Py_HUGE_VAL * 0.) +# else /* __INTEL_COMPILER */ +# if defined(ICC_NAN_STRICT) #pragma float_control(push) #pragma float_control(precise, on) #pragma float_control(except, on) @@ -161,12 +130,12 @@ PyAPI_FUNC(double) _Py_force_double(double); return sqrt(-1.0); } #pragma float_control (pop) - #define Py_NAN __icc_nan() - #else /* ICC_NAN_RELAXED as default for Intel Compiler */ +# define Py_NAN __icc_nan() +# else /* ICC_NAN_RELAXED as default for Intel Compiler */ static const union { unsigned char buf[8]; double __icc_nan; } __nan_store = {0,0,0,0,0,0,0xf8,0x7f}; - #define Py_NAN (__nan_store.__icc_nan) - #endif /* ICC_NAN_STRICT */ -#endif /* __INTEL_COMPILER */ +# define Py_NAN (__nan_store.__icc_nan) +# endif /* ICC_NAN_STRICT */ +# endif /* __INTEL_COMPILER */ #endif #endif /* Py_PYMATH_H */ From webhook-mailer at python.org Mon Oct 11 18:18:34 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 22:18:34 -0000 Subject: [Python-checkins] bpo-45439: _PyObject_Call() only checks tp_vectorcall_offset once (GH-28890) Message-ID: https://github.com/python/cpython/commit/fb8f208a4ddb38eedee71f9ecd0f22058802dab1 commit: fb8f208a4ddb38eedee71f9ecd0f22058802dab1 branch: main author: Victor Stinner committer: vstinner date: 2021-10-12T00:18:26+02:00 summary: bpo-45439: _PyObject_Call() only checks tp_vectorcall_offset once (GH-28890) Add _PyVectorcall_Call() helper function. Add "assert(PyCallable_Check(callable));" to PyVectorcall_Call(), similar check than PyVectorcall_Function(). files: M Include/cpython/abstract.h M Objects/call.c diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index db85021964528..ea269f7320405 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -71,6 +71,7 @@ PyVectorcall_Function(PyObject *callable) return NULL; } assert(PyCallable_Check(callable)); + offset = tp->tp_vectorcall_offset; assert(offset > 0); memcpy(&ptr, (char *) callable + offset, sizeof(ptr)); diff --git a/Objects/call.c b/Objects/call.c index 960c37e1961f0..ecf6e68a2c8ee 100644 --- a/Objects/call.c +++ b/Objects/call.c @@ -225,28 +225,11 @@ _PyObject_MakeTpCall(PyThreadState *tstate, PyObject *callable, } -PyObject * -PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) +static PyObject * +_PyVectorcall_Call(PyThreadState *tstate, vectorcallfunc func, + PyObject *callable, PyObject *tuple, PyObject *kwargs) { - PyThreadState *tstate = _PyThreadState_GET(); - vectorcallfunc func; - - /* get vectorcallfunc as in PyVectorcall_Function, but without - * the Py_TPFLAGS_HAVE_VECTORCALL check */ - Py_ssize_t offset = Py_TYPE(callable)->tp_vectorcall_offset; - if (offset <= 0) { - _PyErr_Format(tstate, PyExc_TypeError, - "'%.200s' object does not support vectorcall", - Py_TYPE(callable)->tp_name); - return NULL; - } - memcpy(&func, (char *) callable + offset, sizeof(func)); - if (func == NULL) { - _PyErr_Format(tstate, PyExc_TypeError, - "'%.200s' object does not support vectorcall", - Py_TYPE(callable)->tp_name); - return NULL; - } + assert(func != NULL); Py_ssize_t nargs = PyTuple_GET_SIZE(tuple); @@ -272,6 +255,35 @@ PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) } +PyObject * +PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) +{ + PyThreadState *tstate = _PyThreadState_GET(); + + /* get vectorcallfunc as in PyVectorcall_Function, but without + * the Py_TPFLAGS_HAVE_VECTORCALL check */ + Py_ssize_t offset = Py_TYPE(callable)->tp_vectorcall_offset; + if (offset <= 0) { + _PyErr_Format(tstate, PyExc_TypeError, + "'%.200s' object does not support vectorcall", + Py_TYPE(callable)->tp_name); + return NULL; + } + assert(PyCallable_Check(callable)); + + vectorcallfunc func; + memcpy(&func, (char *) callable + offset, sizeof(func)); + if (func == NULL) { + _PyErr_Format(tstate, PyExc_TypeError, + "'%.200s' object does not support vectorcall", + Py_TYPE(callable)->tp_name); + return NULL; + } + + return _PyVectorcall_Call(tstate, func, callable, tuple, kwargs); +} + + PyObject * _PyObject_Call(PyThreadState *tstate, PyObject *callable, PyObject *args, PyObject *kwargs) @@ -286,8 +298,9 @@ _PyObject_Call(PyThreadState *tstate, PyObject *callable, assert(PyTuple_Check(args)); assert(kwargs == NULL || PyDict_Check(kwargs)); - if (PyVectorcall_Function(callable) != NULL) { - return PyVectorcall_Call(callable, args, kwargs); + vectorcallfunc vector_func = PyVectorcall_Function(callable); + if (vector_func != NULL) { + return _PyVectorcall_Call(tstate, vector_func, callable, args, kwargs); } else { call = Py_TYPE(callable)->tp_call; From webhook-mailer at python.org Mon Oct 11 18:42:27 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 22:42:27 -0000 Subject: [Python-checkins] bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891) Message-ID: https://github.com/python/cpython/commit/ce3489cfdb9f0e050bdc45ce5d3902c2577ea683 commit: ce3489cfdb9f0e050bdc45ce5d3902c2577ea683 branch: main author: Victor Stinner committer: vstinner date: 2021-10-12T00:42:23+02:00 summary: bpo-45439: Rename _PyObject_CallNoArg() to _PyObject_CallNoArgs() (GH-28891) Fix typo in the private _PyObject_CallNoArg() function name: rename it to _PyObject_CallNoArgs() to be consistent with the public function PyObject_CallNoArgs(). files: M Include/cpython/abstract.h M Include/internal/pycore_call.h M Modules/_collectionsmodule.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/cfield.c M Modules/_ctypes/stgdict.c M Modules/_functoolsmodule.c M Modules/_io/bufferedio.c M Modules/_lsprof.c M Modules/_sqlite/connection.c M Modules/_ssl.c M Modules/_testcapimodule.c M Modules/itertoolsmodule.c M Modules/main.c M Modules/mathmodule.c M Modules/posixmodule.c M Objects/abstract.c M Objects/bytesobject.c M Objects/call.c M Objects/complexobject.c M Objects/dictobject.c M Objects/enumobject.c M Objects/fileobject.c M Objects/genobject.c M Objects/iterobject.c M Objects/moduleobject.c M Objects/object.c M Objects/odictobject.c M Objects/typeobject.c M Parser/tokenizer.c M Python/bltinmodule.c M Python/ceval.c M Python/codecs.c M Python/errors.c M Python/marshal.c M Python/sysmodule.c diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index ea269f7320405..18bf0137639a9 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -164,7 +164,7 @@ _PyObject_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs) Private static inline function variant of public function PyObject_CallNoArgs(). */ static inline PyObject * -_PyObject_CallNoArg(PyObject *func) { +_PyObject_CallNoArgs(PyObject *func) { PyThreadState *tstate = PyThreadState_Get(); return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } diff --git a/Include/internal/pycore_call.h b/Include/internal/pycore_call.h index f7d856a54bd4a..2f7c07a4b411a 100644 --- a/Include/internal/pycore_call.h +++ b/Include/internal/pycore_call.h @@ -29,7 +29,7 @@ PyAPI_FUNC(PyObject *) _PyObject_Call( PyObject *kwargs); static inline PyObject * -_PyObject_CallNoArgTstate(PyThreadState *tstate, PyObject *func) { +_PyObject_CallNoArgsTstate(PyThreadState *tstate, PyObject *func) { return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 01522cd494587..8a690b46fa3e5 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -2010,7 +2010,7 @@ defdict_missing(defdictobject *dd, PyObject *key) Py_DECREF(tup); return NULL; } - value = _PyObject_CallNoArg(factory); + value = _PyObject_CallNoArgs(factory); if (value == NULL) return value; if (PyObject_SetItem((PyObject *)dd, key, value) < 0) { diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 1d6358c910982..7ff101e571490 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -216,7 +216,7 @@ PyDict_SetItemProxy(PyObject *dict, PyObject *key, PyObject *item) PyObject *proxy; int result; - obj = _PyObject_CallNoArg((PyObject *)&DictRemover_Type); + obj = _PyObject_CallNoArgs((PyObject *)&DictRemover_Type); if (obj == NULL) return -1; @@ -506,7 +506,7 @@ StructUnionType_new(PyTypeObject *type, PyObject *args, PyObject *kwds, int isSt return NULL; } - dict = (StgDictObject *)_PyObject_CallNoArg((PyObject *)&PyCStgDict_Type); + dict = (StgDictObject *)_PyObject_CallNoArgs((PyObject *)&PyCStgDict_Type); if (!dict) { Py_DECREF(result); return NULL; @@ -1067,7 +1067,7 @@ PyCPointerType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) stgdict items size, align, length contain info about pointers itself, stgdict->proto has info about the pointed to type! */ - stgdict = (StgDictObject *)_PyObject_CallNoArg( + stgdict = (StgDictObject *)_PyObject_CallNoArgs( (PyObject *)&PyCStgDict_Type); if (!stgdict) return NULL; @@ -1550,7 +1550,7 @@ PyCArrayType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) goto error; } - stgdict = (StgDictObject *)_PyObject_CallNoArg( + stgdict = (StgDictObject *)_PyObject_CallNoArgs( (PyObject *)&PyCStgDict_Type); if (!stgdict) goto error; @@ -2006,7 +2006,7 @@ static PyObject *CreateSwappedType(PyTypeObject *type, PyObject *args, PyObject if (result == NULL) return NULL; - stgdict = (StgDictObject *)_PyObject_CallNoArg( + stgdict = (StgDictObject *)_PyObject_CallNoArgs( (PyObject *)&PyCStgDict_Type); if (!stgdict) { Py_DECREF(result); @@ -2120,7 +2120,7 @@ PyCSimpleType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) goto error; } - stgdict = (StgDictObject *)_PyObject_CallNoArg( + stgdict = (StgDictObject *)_PyObject_CallNoArgs( (PyObject *)&PyCStgDict_Type); if (!stgdict) goto error; @@ -2560,7 +2560,7 @@ PyCFuncPtrType_new(PyTypeObject *type, PyObject *args, PyObject *kwds) PyTypeObject *result; StgDictObject *stgdict; - stgdict = (StgDictObject *)_PyObject_CallNoArg( + stgdict = (StgDictObject *)_PyObject_CallNoArgs( (PyObject *)&PyCStgDict_Type); if (!stgdict) return NULL; @@ -3985,10 +3985,10 @@ _build_callargs(PyCFuncPtrObject *self, PyObject *argtypes, goto error; } if (PyCArrayTypeObject_Check(ob)) - ob = _PyObject_CallNoArg(ob); + ob = _PyObject_CallNoArgs(ob); else /* Create an instance of the pointed-to type */ - ob = _PyObject_CallNoArg(dict->proto); + ob = _PyObject_CallNoArgs(dict->proto); /* XXX Is the following correct any longer? We must not pass a byref() to the array then but @@ -5596,7 +5596,7 @@ cast(void *ptr, PyObject *src, PyObject *ctype) CDataObject *result; if (0 == cast_check_pointertype(ctype)) return NULL; - result = (CDataObject *)_PyObject_CallNoArg(ctype); + result = (CDataObject *)_PyObject_CallNoArgs(ctype); if (result == NULL) return NULL; diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 5a4d1c543f100..16afb240a54ed 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -187,7 +187,7 @@ static void _CallPythonObject(void *mem, */ } else if (dict) { /* Hm, shouldn't we use PyCData_AtAddress() or something like that instead? */ - CDataObject *obj = (CDataObject *)_PyObject_CallNoArg(cnv); + CDataObject *obj = (CDataObject *)_PyObject_CallNoArgs(cnv); if (!obj) { PrintError("create argument %zd:\n", i); Py_DECREF(cnv); @@ -572,7 +572,7 @@ long Call_CanUnloadNow(void) return E_FAIL; } - result = _PyObject_CallNoArg(func); + result = _PyObject_CallNoArgs(func); Py_DECREF(func); if (!result) { PyErr_WriteUnraisable(context ? context : Py_None); diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index ec6feca8b0f80..2909f2f0f7dc4 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -61,7 +61,7 @@ PyCField_FromDesc(PyObject *desc, Py_ssize_t index, #define CONT_BITFIELD 2 #define EXPAND_BITFIELD 3 - self = (CFieldObject *)_PyObject_CallNoArg((PyObject *)&PyCField_Type); + self = (CFieldObject *)_PyObject_CallNoArgs((PyObject *)&PyCField_Type); if (self == NULL) return NULL; dict = PyType_stgdict(desc); diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c index 747339dee352e..d44b54cd55280 100644 --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -248,7 +248,7 @@ MakeFields(PyObject *type, CFieldObject *descr, } continue; } - new_descr = (CFieldObject *)_PyObject_CallNoArg((PyObject *)&PyCField_Type); + new_descr = (CFieldObject *)_PyObject_CallNoArgs((PyObject *)&PyCField_Type); if (new_descr == NULL) { Py_DECREF(fdescr); Py_DECREF(fieldlist); diff --git a/Modules/_functoolsmodule.c b/Modules/_functoolsmodule.c index d6dfefdd87767..ed60ae3d7ea36 100644 --- a/Modules/_functoolsmodule.c +++ b/Modules/_functoolsmodule.c @@ -1440,7 +1440,7 @@ static int _functools_exec(PyObject *module) { _functools_state *state = get_functools_state(module); - state->kwd_mark = _PyObject_CallNoArg((PyObject *)&PyBaseObject_Type); + state->kwd_mark = _PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); if (state->kwd_mark == NULL) { return -1; } diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index ba966f568b399..74d4fb5da70aa 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -1550,7 +1550,7 @@ _bufferedreader_read_all(buffered *self) goto cleanup; } if (readall) { - tmp = _PyObject_CallNoArg(readall); + tmp = _PyObject_CallNoArgs(readall); Py_DECREF(readall); if (tmp == NULL) goto cleanup; diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 703067cc743a5..0ca3f1e1a2533 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -76,7 +76,7 @@ _lsprof_get_state(PyObject *module) static _PyTime_t CallExternalTimer(ProfilerObject *pObj) { - PyObject *o = _PyObject_CallNoArg(pObj->externalTimer); + PyObject *o = _PyObject_CallNoArgs(pObj->externalTimer); if (o == NULL) { PyErr_WriteUnraisable(pObj->externalTimer); return 0; diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 02481d9fbd8bb..a331757562aa0 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -710,7 +710,7 @@ step_callback(sqlite3_context *context, int argc, sqlite3_value **params) if (*aggregate_instance == NULL) { callback_context *ctx = (callback_context *)sqlite3_user_data(context); assert(ctx != NULL); - *aggregate_instance = _PyObject_CallNoArg(ctx->callable); + *aggregate_instance = _PyObject_CallNoArgs(ctx->callable); if (!*aggregate_instance) { set_sqlite_error(context, "user-defined aggregate's '__init__' method raised error"); @@ -1008,7 +1008,7 @@ progress_callback(void *ctx) assert(ctx != NULL); PyObject *callable = ((callback_context *)ctx)->callable; - ret = _PyObject_CallNoArg(callable); + ret = _PyObject_CallNoArgs(callable); if (!ret) { /* abort query if error occurred */ rc = -1; diff --git a/Modules/_ssl.c b/Modules/_ssl.c index fedb35b5778b3..c163ce69d5ee9 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -3773,7 +3773,7 @@ _password_callback(char *buf, int size, int rwflag, void *userdata) } if (pw_info->callable) { - fn_ret = _PyObject_CallNoArg(pw_info->callable); + fn_ret = _PyObject_CallNoArgs(pw_info->callable); if (!fn_ret) { /* TODO: It would be nice to move _ctypes_add_traceback() into the core python API, so we could use it to add a frame here */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 4a2907f3f06bd..775464c2dee43 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2866,7 +2866,7 @@ _make_call(void *callable) PyObject *rc; int success; PyGILState_STATE s = PyGILState_Ensure(); - rc = _PyObject_CallNoArg((PyObject *)callable); + rc = _PyObject_CallNoArgs((PyObject *)callable); success = (rc != NULL); Py_XDECREF(rc); PyGILState_Release(s); @@ -2937,7 +2937,7 @@ static int _pending_callback(void *arg) { /* we assume the argument is callable object to which we own a reference */ PyObject *callable = (PyObject *)arg; - PyObject *r = _PyObject_CallNoArg(callable); + PyObject *r = _PyObject_CallNoArgs(callable); Py_DECREF(callable); Py_XDECREF(r); return r != NULL ? 0 : -1; @@ -3729,7 +3729,7 @@ slot_tp_del(PyObject *self) /* Execute __del__ method, if any. */ del = _PyObject_LookupSpecial(self, &PyId___tp_del__); if (del != NULL) { - res = _PyObject_CallNoArg(del); + res = _PyObject_CallNoArgs(del); if (res == NULL) PyErr_WriteUnraisable(del); else @@ -4358,7 +4358,7 @@ temporary_c_thread(void *data) /* Allocate a Python thread state for this thread */ state = PyGILState_Ensure(); - res = _PyObject_CallNoArg(test_c_thread->callback); + res = _PyObject_CallNoArgs(test_c_thread->callback); Py_CLEAR(test_c_thread->callback); if (res == NULL) { @@ -4893,7 +4893,7 @@ check_pyobject_freed_is_freed(PyObject *self, PyObject *Py_UNUSED(args)) #ifdef _Py_ADDRESS_SANITIZER Py_RETURN_NONE; #else - PyObject *op = _PyObject_CallNoArg((PyObject *)&PyBaseObject_Type); + PyObject *op = _PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); if (op == NULL) { return NULL; } @@ -5271,7 +5271,7 @@ bad_get(PyObject *module, PyObject *const *args, Py_ssize_t nargs) return NULL; } - PyObject *res = _PyObject_CallNoArg(cls); + PyObject *res = _PyObject_CallNoArgs(cls); if (res == NULL) { return NULL; } diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c index 0df179673a111..d6f898cdbf18d 100644 --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -1058,7 +1058,7 @@ itertools_tee_impl(PyObject *module, PyObject *iterable, Py_ssize_t n) PyTuple_SET_ITEM(result, 0, copyable); for (i = 1; i < n; i++) { - copyable = _PyObject_CallNoArg(copyfunc); + copyable = _PyObject_CallNoArgs(copyfunc); if (copyable == NULL) { Py_DECREF(copyfunc); Py_DECREF(result); diff --git a/Modules/main.c b/Modules/main.c index 2684d230672b9..39b8c71071e09 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -457,7 +457,7 @@ pymain_run_interactive_hook(int *exitcode) goto error; } - result = _PyObject_CallNoArg(hook); + result = _PyObject_CallNoArgs(hook); Py_DECREF(hook); if (result == NULL) { goto error; diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 5e9f63f6c6e02..62aceced435a0 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -1206,7 +1206,7 @@ math_ceil(PyObject *module, PyObject *number) if (!PyFloat_CheckExact(number)) { PyObject *method = _PyObject_LookupSpecial(number, &PyId___ceil__); if (method != NULL) { - PyObject *result = _PyObject_CallNoArg(method); + PyObject *result = _PyObject_CallNoArgs(method); Py_DECREF(method); return result; } @@ -1275,7 +1275,7 @@ math_floor(PyObject *module, PyObject *number) { PyObject *method = _PyObject_LookupSpecial(number, &PyId___floor__); if (method != NULL) { - PyObject *result = _PyObject_CallNoArg(method); + PyObject *result = _PyObject_CallNoArgs(method); Py_DECREF(method); return result; } @@ -2130,7 +2130,7 @@ math_trunc(PyObject *module, PyObject *x) Py_TYPE(x)->tp_name); return NULL; } - result = _PyObject_CallNoArg(trunc); + result = _PyObject_CallNoArgs(trunc); Py_DECREF(trunc); return result; } diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index bd38bea4c6c99..01012c181b423 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -561,7 +561,7 @@ run_at_forkers(PyObject *lst, int reverse) for (i = 0; i < PyList_GET_SIZE(cpy); i++) { PyObject *func, *res; func = PyList_GET_ITEM(cpy, i); - res = _PyObject_CallNoArg(func); + res = _PyObject_CallNoArgs(func); if (res == NULL) PyErr_WriteUnraisable(func); else @@ -1183,7 +1183,7 @@ path_converter(PyObject *o, void *p) if (NULL == func) { goto error_format; } - res = _PyObject_CallNoArg(func); + res = _PyObject_CallNoArgs(func); Py_DECREF(func); if (NULL == res) { goto error_exit; @@ -14380,7 +14380,7 @@ PyOS_FSPath(PyObject *path) _PyType_Name(Py_TYPE(path))); } - path_repr = _PyObject_CallNoArg(func); + path_repr = _PyObject_CallNoArgs(func); Py_DECREF(func); if (NULL == path_repr) { return NULL; diff --git a/Objects/abstract.c b/Objects/abstract.c index c056c6698850c..3312e264ea1c2 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -114,7 +114,7 @@ PyObject_LengthHint(PyObject *o, Py_ssize_t defaultvalue) } return defaultvalue; } - result = _PyObject_CallNoArg(hint); + result = _PyObject_CallNoArgs(hint); Py_DECREF(hint); if (result == NULL) { PyThreadState *tstate = _PyThreadState_GET(); @@ -1576,7 +1576,7 @@ PyNumber_Long(PyObject *o) } trunc_func = _PyObject_LookupSpecial(o, &PyId___trunc__); if (trunc_func) { - result = _PyObject_CallNoArg(trunc_func); + result = _PyObject_CallNoArgs(trunc_func); Py_DECREF(trunc_func); if (result == NULL || PyLong_CheckExact(result)) { return result; diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 27f766bef6ed8..0dfded24b3519 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -572,7 +572,7 @@ format_obj(PyObject *v, const char **pbuf, Py_ssize_t *plen) /* does it support __bytes__? */ func = _PyObject_LookupSpecial(v, &PyId___bytes__); if (func != NULL) { - result = _PyObject_CallNoArg(func); + result = _PyObject_CallNoArgs(func); Py_DECREF(func); if (result == NULL) return NULL; @@ -2622,7 +2622,7 @@ bytes_new_impl(PyTypeObject *type, PyObject *x, const char *encoding, integer argument before deferring to PyBytes_FromObject, something PyObject_Bytes doesn't do. */ else if ((func = _PyObject_LookupSpecial(x, &PyId___bytes__)) != NULL) { - bytes = _PyObject_CallNoArg(func); + bytes = _PyObject_CallNoArgs(func); Py_DECREF(func); if (bytes == NULL) return NULL; diff --git a/Objects/call.c b/Objects/call.c index ecf6e68a2c8ee..cfcd4220f3cf7 100644 --- a/Objects/call.c +++ b/Objects/call.c @@ -1,5 +1,5 @@ #include "Python.h" -#include "pycore_call.h" // _PyObject_CallNoArgTstate() +#include "pycore_call.h" // _PyObject_CallNoArgsTstate() #include "pycore_ceval.h" // _PyEval_EvalFrame() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_pyerrors.h" // _PyErr_Occurred() @@ -110,7 +110,7 @@ PyObject * PyObject_CallNoArgs(PyObject *func) { PyThreadState *tstate = _PyThreadState_GET(); - return _PyObject_CallNoArgTstate(tstate, func); + return _PyObject_CallNoArgsTstate(tstate, func); } @@ -402,7 +402,7 @@ PyObject_CallObject(PyObject *callable, PyObject *args) PyThreadState *tstate = _PyThreadState_GET(); assert(!_PyErr_Occurred(tstate)); if (args == NULL) { - return _PyObject_CallNoArgTstate(tstate, callable); + return _PyObject_CallNoArgsTstate(tstate, callable); } if (!PyTuple_Check(args)) { _PyErr_SetString(tstate, PyExc_TypeError, @@ -468,7 +468,7 @@ _PyObject_CallFunctionVa(PyThreadState *tstate, PyObject *callable, } if (!format || !*format) { - return _PyObject_CallNoArgTstate(tstate, callable); + return _PyObject_CallNoArgsTstate(tstate, callable); } if (is_size_t) { diff --git a/Objects/complexobject.c b/Objects/complexobject.c index f08f03fcb49a5..ff8fcba20391b 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -284,7 +284,7 @@ try_complex_special_method(PyObject *op) f = _PyObject_LookupSpecial(op, &PyId___complex__); if (f) { - PyObject *res = _PyObject_CallNoArg(f); + PyObject *res = _PyObject_CallNoArgs(f); Py_DECREF(f); if (!res || PyComplex_CheckExact(res)) { return res; diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 824cba949d7a8..f883ca7c5b5c6 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -1897,7 +1897,7 @@ _PyDict_FromKeys(PyObject *cls, PyObject *iterable, PyObject *value) PyObject *d; int status; - d = _PyObject_CallNoArg(cls); + d = _PyObject_CallNoArgs(cls); if (d == NULL) return NULL; diff --git a/Objects/enumobject.c b/Objects/enumobject.c index 0ebaf5ea1fc79..aae7dffec1ba0 100644 --- a/Objects/enumobject.c +++ b/Objects/enumobject.c @@ -298,7 +298,7 @@ reversed_new_impl(PyTypeObject *type, PyObject *seq) return NULL; } if (reversed_meth != NULL) { - PyObject *res = _PyObject_CallNoArg(reversed_meth); + PyObject *res = _PyObject_CallNoArgs(reversed_meth); Py_DECREF(reversed_meth); return res; } diff --git a/Objects/fileobject.c b/Objects/fileobject.c index 5a2816f55244b..5cb017e8e206e 100644 --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -190,7 +190,7 @@ PyObject_AsFileDescriptor(PyObject *o) return -1; } else if (meth != NULL) { - PyObject *fno = _PyObject_CallNoArg(meth); + PyObject *fno = _PyObject_CallNoArgs(meth); Py_DECREF(meth); if (fno == NULL) return -1; diff --git a/Objects/genobject.c b/Objects/genobject.c index 8bd6c8d2c4ccc..8fa04e8d68413 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -319,7 +319,7 @@ gen_close_iter(PyObject *yf) PyErr_WriteUnraisable(yf); } if (meth) { - retval = _PyObject_CallNoArg(meth); + retval = _PyObject_CallNoArgs(meth); Py_DECREF(meth); if (retval == NULL) return -1; diff --git a/Objects/iterobject.c b/Objects/iterobject.c index e493e41131b70..acfc539fae0b7 100644 --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -217,7 +217,7 @@ calliter_iternext(calliterobject *it) return NULL; } - result = _PyObject_CallNoArg(it->it_callable); + result = _PyObject_CallNoArgs(it->it_callable); if (result != NULL) { int ok; diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index 1c37eb08c89a3..f008995f32f03 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -866,7 +866,7 @@ module_dir(PyObject *self, PyObject *args) if (PyDict_Check(dict)) { PyObject *dirfunc = _PyDict_GetItemIdWithError(dict, &PyId___dir__); if (dirfunc) { - result = _PyObject_CallNoArg(dirfunc); + result = _PyObject_CallNoArgs(dirfunc); } else if (!PyErr_Occurred()) { result = PyDict_Keys(dict); diff --git a/Objects/object.c b/Objects/object.c index aa84815e56e86..d269274c74ec0 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -560,7 +560,7 @@ PyObject_Bytes(PyObject *v) func = _PyObject_LookupSpecial(v, &PyId___bytes__); if (func != NULL) { - result = _PyObject_CallNoArg(func); + result = _PyObject_CallNoArgs(func); Py_DECREF(func); if (result == NULL) return NULL; @@ -1521,7 +1521,7 @@ _dir_object(PyObject *obj) return NULL; } /* use __dir__ */ - result = _PyObject_CallNoArg(dirfunc); + result = _PyObject_CallNoArgs(dirfunc); Py_DECREF(dirfunc); if (result == NULL) return NULL; diff --git a/Objects/odictobject.c b/Objects/odictobject.c index e5361da6dcded..1462a3b397752 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -1203,7 +1203,7 @@ odict_copy(register PyODictObject *od, PyObject *Py_UNUSED(ignored)) if (PyODict_CheckExact(od)) od_copy = PyODict_New(); else - od_copy = _PyObject_CallNoArg((PyObject *)Py_TYPE(od)); + od_copy = _PyObject_CallNoArgs((PyObject *)Py_TYPE(od)); if (od_copy == NULL) return NULL; @@ -2221,7 +2221,7 @@ mutablemapping_update_arg(PyObject *self, PyObject *arg) return -1; } if (func != NULL) { - PyObject *keys = _PyObject_CallNoArg(func); + PyObject *keys = _PyObject_CallNoArgs(func); Py_DECREF(func); if (keys == NULL) { return -1; @@ -2253,7 +2253,7 @@ mutablemapping_update_arg(PyObject *self, PyObject *arg) return -1; } if (func != NULL) { - PyObject *items = _PyObject_CallNoArg(func); + PyObject *items = _PyObject_CallNoArgs(func); Py_DECREF(func); if (items == NULL) { return -1; diff --git a/Objects/typeobject.c b/Objects/typeobject.c index a81d071e31d23..a73a696a1ec88 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -1640,7 +1640,7 @@ call_unbound_noarg(int unbound, PyObject *func, PyObject *self) return PyObject_CallOneArg(func, self); } else { - return _PyObject_CallNoArg(func); + return _PyObject_CallNoArgs(func); } } @@ -5019,7 +5019,7 @@ _PyObject_GetState(PyObject *obj, int required) Py_DECREF(slotnames); } else { /* getstate != NULL */ - state = _PyObject_CallNoArg(getstate); + state = _PyObject_CallNoArgs(getstate); Py_DECREF(getstate); if (state == NULL) return NULL; @@ -5044,7 +5044,7 @@ _PyObject_GetNewArguments(PyObject *obj, PyObject **args, PyObject **kwargs) __getnewargs_ex__ on the object. */ getnewargs_ex = _PyObject_LookupSpecial(obj, &PyId___getnewargs_ex__); if (getnewargs_ex != NULL) { - PyObject *newargs = _PyObject_CallNoArg(getnewargs_ex); + PyObject *newargs = _PyObject_CallNoArgs(getnewargs_ex); Py_DECREF(getnewargs_ex); if (newargs == NULL) { return -1; @@ -5097,7 +5097,7 @@ _PyObject_GetNewArguments(PyObject *obj, PyObject **args, PyObject **kwargs) __getnewargs__ instead. */ getnewargs = _PyObject_LookupSpecial(obj, &PyId___getnewargs__); if (getnewargs != NULL) { - *args = _PyObject_CallNoArg(getnewargs); + *args = _PyObject_CallNoArgs(getnewargs); Py_DECREF(getnewargs); if (*args == NULL) { return -1; @@ -5358,7 +5358,7 @@ object___reduce_ex___impl(PyObject *self, int protocol) override = (clsreduce != objreduce); Py_DECREF(clsreduce); if (override) { - res = _PyObject_CallNoArg(reduce); + res = _PyObject_CallNoArgs(reduce); Py_DECREF(reduce); return res; } diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 90eee28e12653..0e3caf3086ad9 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -477,7 +477,7 @@ fp_setreadl(struct tok_state *tok, const char* enc) Py_XSETREF(tok->decoding_readline, readline); if (pos > 0) { - PyObject *bufobj = _PyObject_CallNoArg(readline); + PyObject *bufobj = _PyObject_CallNoArgs(readline); if (bufobj == NULL) return 0; Py_DECREF(bufobj); diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index 1f038166890fd..f91bb9ffdbcf3 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -2305,7 +2305,7 @@ builtin_round_impl(PyObject *module, PyObject *number, PyObject *ndigits) } if (ndigits == Py_None) - result = _PyObject_CallNoArg(round); + result = _PyObject_CallNoArgs(round); else result = PyObject_CallOneArg(round, ndigits); Py_DECREF(round); diff --git a/Python/ceval.c b/Python/ceval.c index a63b80395dcfc..e9a9c1dccd82c 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -867,7 +867,7 @@ match_keys(PyThreadState *tstate, PyObject *map, PyObject *keys) goto fail; } // dummy = object() - dummy = _PyObject_CallNoArg((PyObject *)&PyBaseObject_Type); + dummy = _PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); if (dummy == NULL) { goto fail; } @@ -4295,7 +4295,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } SET_TOP(exit); Py_DECREF(mgr); - res = _PyObject_CallNoArg(enter); + res = _PyObject_CallNoArgs(enter); Py_DECREF(enter); if (res == NULL) goto error; @@ -4333,7 +4333,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } SET_TOP(exit); Py_DECREF(mgr); - res = _PyObject_CallNoArg(enter); + res = _PyObject_CallNoArgs(enter); Py_DECREF(enter); if (res == NULL) { goto error; @@ -5859,7 +5859,7 @@ do_raise(PyThreadState *tstate, PyObject *exc, PyObject *cause) if (PyExceptionClass_Check(exc)) { type = exc; - value = _PyObject_CallNoArg(exc); + value = _PyObject_CallNoArgs(exc); if (value == NULL) goto raise_error; if (!PyExceptionInstance_Check(value)) { @@ -5890,7 +5890,7 @@ do_raise(PyThreadState *tstate, PyObject *exc, PyObject *cause) if (cause) { PyObject *fixed_cause; if (PyExceptionClass_Check(cause)) { - fixed_cause = _PyObject_CallNoArg(cause); + fixed_cause = _PyObject_CallNoArgs(cause); if (fixed_cause == NULL) goto raise_error; Py_DECREF(cause); diff --git a/Python/codecs.c b/Python/codecs.c index fa329ce243642..9ee566bd0f761 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -280,7 +280,7 @@ PyObject *codec_makeincrementalcodec(PyObject *codec_info, if (errors) ret = PyObject_CallFunction(inccodec, "s", errors); else - ret = _PyObject_CallNoArg(inccodec); + ret = _PyObject_CallNoArgs(inccodec); Py_DECREF(inccodec); return ret; } diff --git a/Python/errors.c b/Python/errors.c index b2030f728a7eb..36a529e65a170 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -91,7 +91,7 @@ _PyErr_CreateException(PyObject *exception_type, PyObject *value) PyObject *exc; if (value == NULL || value == Py_None) { - exc = _PyObject_CallNoArg(exception_type); + exc = _PyObject_CallNoArgs(exception_type); } else if (PyTuple_Check(value)) { exc = PyObject_Call(exception_type, value, NULL); diff --git a/Python/marshal.c b/Python/marshal.c index 346384edea618..530c8d0f23857 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -1292,7 +1292,7 @@ r_object(RFILE *p) if (n == 0 && type == TYPE_FROZENSET) { /* call frozenset() to get the empty frozenset singleton */ - v = _PyObject_CallNoArg((PyObject*)&PyFrozenSet_Type); + v = _PyObject_CallNoArgs((PyObject*)&PyFrozenSet_Type); if (v == NULL) break; R_REF(v); diff --git a/Python/sysmodule.c b/Python/sysmodule.c index f2cd3a9d5a010..0154af0b76fba 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1676,7 +1676,7 @@ _PySys_GetSizeOf(PyObject *o) } } else { - res = _PyObject_CallNoArg(method); + res = _PyObject_CallNoArgs(method); Py_DECREF(method); } From webhook-mailer at python.org Mon Oct 11 19:24:11 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 11 Oct 2021 23:24:11 -0000 Subject: [Python-checkins] bpo-45433: Do not link libpython against libcrypt (GH-28881) Message-ID: https://github.com/python/cpython/commit/be21706f3760bec8bd11f85ce02ed6792b07f51f commit: be21706f3760bec8bd11f85ce02ed6792b07f51f branch: main author: Mike Gilbert committer: vstinner date: 2021-10-12T01:24:03+02:00 summary: bpo-45433: Do not link libpython against libcrypt (GH-28881) Save/restore LIBS when calling AC_SEARCH_LIBS(..., crypt). This avoid linking libpython with libcrypt. files: A Misc/NEWS.d/next/Build/2021-10-11-16-08-37.bpo-45433.pVDkMV.rst M Doc/whatsnew/3.11.rst M configure M configure.ac diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 0d30fe8b64644..21a46b4b72133 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -457,6 +457,8 @@ Build Changes * CPython can now be built with the ThinLTO option via ``--with-lto=thin``. (Contributed by Dong-hee Na and Brett Holman in :issue:`44340`.) +* libpython is no longer linked against libcrypt. + (Contributed by Mike Gilbert in :issue:`45433`.) C API Changes ============= diff --git a/Misc/NEWS.d/next/Build/2021-10-11-16-08-37.bpo-45433.pVDkMV.rst b/Misc/NEWS.d/next/Build/2021-10-11-16-08-37.bpo-45433.pVDkMV.rst new file mode 100644 index 0000000000000..27869f08fbb9b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-11-16-08-37.bpo-45433.pVDkMV.rst @@ -0,0 +1 @@ +Avoid linking libpython with libcrypt. diff --git a/configure b/configure index 15c7c54b09536..70f28b0c7064e 100755 --- a/configure +++ b/configure @@ -13227,6 +13227,8 @@ done # We search for both crypt and crypt_r as one or the other may be defined # This gets us our -lcrypt in LIBS when required on the target platform. +# Save/restore LIBS to avoid linking libpython with libcrypt. +LIBS_SAVE=$LIBS { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing crypt" >&5 $as_echo_n "checking for library containing crypt... " >&6; } if ${ac_cv_search_crypt+:} false; then : @@ -13368,6 +13370,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext fi +LIBS=$LIBS_SAVE for ac_func in clock_gettime do : diff --git a/configure.ac b/configure.ac index 6c65b2914bf66..afdc68363ceaf 100644 --- a/configure.ac +++ b/configure.ac @@ -4085,6 +4085,8 @@ AC_CHECK_FUNCS(setpgrp, # We search for both crypt and crypt_r as one or the other may be defined # This gets us our -lcrypt in LIBS when required on the target platform. +# Save/restore LIBS to avoid linking libpython with libcrypt. +LIBS_SAVE=$LIBS AC_SEARCH_LIBS(crypt, crypt) AC_SEARCH_LIBS(crypt_r, crypt) @@ -4099,6 +4101,7 @@ char *r = crypt_r("", "", &d); [AC_DEFINE(HAVE_CRYPT_R, 1, [Define if you have the crypt_r() function.])], []) ) +LIBS=$LIBS_SAVE AC_CHECK_FUNCS(clock_gettime, [], [ AC_CHECK_LIB(rt, clock_gettime, [ From webhook-mailer at python.org Tue Oct 12 02:38:28 2021 From: webhook-mailer at python.org (vstinner) Date: Tue, 12 Oct 2021 06:38:28 -0000 Subject: [Python-checkins] bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895) Message-ID: https://github.com/python/cpython/commit/d943d19172aa93ce88bade15b9f23a0ce3bc72ff commit: d943d19172aa93ce88bade15b9f23a0ce3bc72ff branch: main author: Victor Stinner committer: vstinner date: 2021-10-12T08:38:19+02:00 summary: bpo-45439: Move _PyObject_CallNoArgs() to pycore_call.h (GH-28895) * Move _PyObject_CallNoArgs() to pycore_call.h (internal C API). * _ssl, _sqlite and _testcapi extensions now call the public PyObject_CallNoArgs() function, rather than _PyObject_CallNoArgs(). * _lsprof extension is now built with Py_BUILD_CORE_MODULE macro defined to get access to internal _PyObject_CallNoArgs(). files: M Include/cpython/abstract.h M Include/internal/pycore_call.h M Modules/_collectionsmodule.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/cfield.c M Modules/_ctypes/stgdict.c M Modules/_functoolsmodule.c M Modules/_io/bufferedio.c M Modules/_lsprof.c M Modules/_sqlite/connection.c M Modules/_ssl.c M Modules/_testcapimodule.c M Modules/itertoolsmodule.c M Modules/main.c M Modules/mathmodule.c M Modules/posixmodule.c M Modules/signalmodule.c M Objects/abstract.c M Objects/bytesobject.c M Objects/complexobject.c M Objects/dictobject.c M Objects/enumobject.c M Objects/fileobject.c M Objects/genobject.c M Objects/iterobject.c M Objects/moduleobject.c M Objects/object.c M Objects/odictobject.c M Objects/typeobject.c M Parser/tokenizer.c M Python/bltinmodule.c M Python/ceval.c M Python/codecs.c M Python/errors.c M Python/marshal.c M Python/sysmodule.c M setup.py diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index 18bf0137639a9..0814bfa62e1a4 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -160,15 +160,6 @@ _PyObject_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs) return _PyObject_FastCallTstate(tstate, func, args, nargs); } -/* Call a callable without any arguments - Private static inline function variant of public function - PyObject_CallNoArgs(). */ -static inline PyObject * -_PyObject_CallNoArgs(PyObject *func) { - PyThreadState *tstate = PyThreadState_Get(); - return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); -} - static inline PyObject * PyObject_CallOneArg(PyObject *func, PyObject *arg) { diff --git a/Include/internal/pycore_call.h b/Include/internal/pycore_call.h index 2f7c07a4b411a..28a4194db7484 100644 --- a/Include/internal/pycore_call.h +++ b/Include/internal/pycore_call.h @@ -33,6 +33,13 @@ _PyObject_CallNoArgsTstate(PyThreadState *tstate, PyObject *func) { return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } +// Private static inline function variant of public PyObject_CallNoArgs() +static inline PyObject * +_PyObject_CallNoArgs(PyObject *func) { + PyThreadState *tstate = PyThreadState_Get(); + return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); +} + #ifdef __cplusplus } #endif diff --git a/Modules/_collectionsmodule.c b/Modules/_collectionsmodule.c index 8a690b46fa3e5..c6de9636032c0 100644 --- a/Modules/_collectionsmodule.c +++ b/Modules/_collectionsmodule.c @@ -1,4 +1,5 @@ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_long.h" // _PyLong_GetZero() #include "structmember.h" // PyMemberDef diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 7ff101e571490..2567067f0b39f 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -102,6 +102,7 @@ bytes(cdata) #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "structmember.h" // PyMemberDef #include diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 16afb240a54ed..18b010493628c 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -1,4 +1,5 @@ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "frameobject.h" #include diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 2909f2f0f7dc4..6788aee78a2d3 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1,5 +1,6 @@ #include "Python.h" #include "pycore_bitutils.h" // _Py_bswap32() +#include "pycore_call.h" // _PyObject_CallNoArgs() #include #ifdef MS_WIN32 diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c index d44b54cd55280..ea3c58b88e704 100644 --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -1,4 +1,5 @@ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include #ifdef MS_WIN32 #include diff --git a/Modules/_functoolsmodule.c b/Modules/_functoolsmodule.c index ed60ae3d7ea36..4c77ee7b7a53c 100644 --- a/Modules/_functoolsmodule.c +++ b/Modules/_functoolsmodule.c @@ -1,4 +1,5 @@ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_object.h" // _PyObject_GC_TRACK diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index 74d4fb5da70aa..dc6371a572b9f 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -9,6 +9,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_object.h" #include "structmember.h" // PyMemberDef #include "_iomodule.h" diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 0ca3f1e1a2533..097d0eff2602d 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -1,4 +1,5 @@ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "rotatingtree.h" /************************************************************/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index a331757562aa0..b4badb111a3c7 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -710,7 +710,7 @@ step_callback(sqlite3_context *context, int argc, sqlite3_value **params) if (*aggregate_instance == NULL) { callback_context *ctx = (callback_context *)sqlite3_user_data(context); assert(ctx != NULL); - *aggregate_instance = _PyObject_CallNoArgs(ctx->callable); + *aggregate_instance = PyObject_CallNoArgs(ctx->callable); if (!*aggregate_instance) { set_sqlite_error(context, "user-defined aggregate's '__init__' method raised error"); @@ -1008,7 +1008,7 @@ progress_callback(void *ctx) assert(ctx != NULL); PyObject *callable = ((callback_context *)ctx)->callable; - ret = _PyObject_CallNoArgs(callable); + ret = PyObject_CallNoArgs(callable); if (!ret) { /* abort query if error occurred */ rc = -1; diff --git a/Modules/_ssl.c b/Modules/_ssl.c index c163ce69d5ee9..b2e241a0a338e 100644 --- a/Modules/_ssl.c +++ b/Modules/_ssl.c @@ -3773,7 +3773,7 @@ _password_callback(char *buf, int size, int rwflag, void *userdata) } if (pw_info->callable) { - fn_ret = _PyObject_CallNoArgs(pw_info->callable); + fn_ret = PyObject_CallNoArgs(pw_info->callable); if (!fn_ret) { /* TODO: It would be nice to move _ctypes_add_traceback() into the core python API, so we could use it to add a frame here */ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 775464c2dee43..7cbd2dc3b6a51 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -2866,7 +2866,7 @@ _make_call(void *callable) PyObject *rc; int success; PyGILState_STATE s = PyGILState_Ensure(); - rc = _PyObject_CallNoArgs((PyObject *)callable); + rc = PyObject_CallNoArgs((PyObject *)callable); success = (rc != NULL); Py_XDECREF(rc); PyGILState_Release(s); @@ -2937,7 +2937,7 @@ static int _pending_callback(void *arg) { /* we assume the argument is callable object to which we own a reference */ PyObject *callable = (PyObject *)arg; - PyObject *r = _PyObject_CallNoArgs(callable); + PyObject *r = PyObject_CallNoArgs(callable); Py_DECREF(callable); Py_XDECREF(r); return r != NULL ? 0 : -1; @@ -3729,7 +3729,7 @@ slot_tp_del(PyObject *self) /* Execute __del__ method, if any. */ del = _PyObject_LookupSpecial(self, &PyId___tp_del__); if (del != NULL) { - res = _PyObject_CallNoArgs(del); + res = PyObject_CallNoArgs(del); if (res == NULL) PyErr_WriteUnraisable(del); else @@ -4358,7 +4358,7 @@ temporary_c_thread(void *data) /* Allocate a Python thread state for this thread */ state = PyGILState_Ensure(); - res = _PyObject_CallNoArgs(test_c_thread->callback); + res = PyObject_CallNoArgs(test_c_thread->callback); Py_CLEAR(test_c_thread->callback); if (res == NULL) { @@ -4893,7 +4893,7 @@ check_pyobject_freed_is_freed(PyObject *self, PyObject *Py_UNUSED(args)) #ifdef _Py_ADDRESS_SANITIZER Py_RETURN_NONE; #else - PyObject *op = _PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); + PyObject *op = PyObject_CallNoArgs((PyObject *)&PyBaseObject_Type); if (op == NULL) { return NULL; } @@ -5271,7 +5271,7 @@ bad_get(PyObject *module, PyObject *const *args, Py_ssize_t nargs) return NULL; } - PyObject *res = _PyObject_CallNoArgs(cls); + PyObject *res = PyObject_CallNoArgs(cls); if (res == NULL) { return NULL; } diff --git a/Modules/itertoolsmodule.c b/Modules/itertoolsmodule.c index d6f898cdbf18d..342a3e6555ba0 100644 --- a/Modules/itertoolsmodule.c +++ b/Modules/itertoolsmodule.c @@ -1,7 +1,6 @@ - - #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_tuple.h" // _PyTuple_ITEMS() diff --git a/Modules/main.c b/Modules/main.c index 39b8c71071e09..c537e6b678515 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -1,6 +1,7 @@ /* Python interpreter main program */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_initconfig.h" // _PyArgv #include "pycore_interp.h" // _PyInterpreterState.sysdict #include "pycore_pathconfig.h" // _PyPathConfig_ComputeSysPath0() diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 62aceced435a0..4fac0cc29e4e9 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -54,7 +54,8 @@ raised for division by zero and mod by zero. #include "Python.h" #include "pycore_bitutils.h" // _Py_bit_length() -#include "pycore_dtoa.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_dtoa.h" // _Py_dg_infinity() #include "pycore_long.h" // _PyLong_GetZero() #include "_math.h" diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 01012c181b423..ada1a5865aae7 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -10,7 +10,8 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pycore_fileutils.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_fileutils.h" // _Py_closerange() #include "pycore_moduleobject.h" // _PyModule_GetState() #ifdef MS_WINDOWS /* include early to avoid conflict with pycore_condvar.h: diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index d5e6a43af2865..6568a4d5aa70e 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -7,7 +7,7 @@ #include "pycore_atomic.h" // _Py_atomic_int #include "pycore_call.h" // _PyObject_Call() #include "pycore_ceval.h" // _PyEval_SignalReceived() -#include "pycore_frame.h" +#include "pycore_frame.h" // InterpreterFrame #include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_pyerrors.h" // _PyErr_SetString() #include "pycore_pylifecycle.h" // NSIG diff --git a/Objects/abstract.c b/Objects/abstract.c index 3312e264ea1c2..0d6cefd3eb861 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2,6 +2,7 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _Py_EnterRecursiveCall() #include "pycore_object.h" // _Py_CheckSlotResult() #include "pycore_pyerrors.h" // _PyErr_Occurred() diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 0dfded24b3519..bc0b075cf405b 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -5,6 +5,7 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_bytes_methods.h" // _Py_bytes_startswith() +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_format.h" // F_LJUST #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_object.h" // _PyObject_GC_TRACK diff --git a/Objects/complexobject.c b/Objects/complexobject.c index ff8fcba20391b..f658dbf336dbf 100644 --- a/Objects/complexobject.c +++ b/Objects/complexobject.c @@ -6,6 +6,7 @@ /* Submitted by Jim Hugunin */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_Init() #include "pycore_pymath.h" // _Py_ADJUST_ERANGE2() diff --git a/Objects/dictobject.c b/Objects/dictobject.c index f883ca7c5b5c6..60470bf66b161 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -112,13 +112,14 @@ As a consequence of this, split keys have a maximum size of 16. #define PyDict_MINSIZE 8 #include "Python.h" -#include "pycore_bitutils.h" // _Py_bit_length -#include "pycore_gc.h" // _PyObject_GC_IS_TRACKED() -#include "pycore_object.h" // _PyObject_GC_TRACK() -#include "pycore_pyerrors.h" // _PyErr_Fetch() -#include "pycore_pystate.h" // _PyThreadState_GET() -#include "pycore_dict.h" -#include "stringlib/eq.h" // unicode_eq() +#include "pycore_bitutils.h" // _Py_bit_length +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_dict.h" // PyDictKeysObject +#include "pycore_gc.h" // _PyObject_GC_IS_TRACKED() +#include "pycore_object.h" // _PyObject_GC_TRACK() +#include "pycore_pyerrors.h" // _PyErr_Fetch() +#include "pycore_pystate.h" // _PyThreadState_GET() +#include "stringlib/eq.h" // unicode_eq() /*[clinic input] class dict "PyDictObject *" "&PyDict_Type" diff --git a/Objects/enumobject.c b/Objects/enumobject.c index aae7dffec1ba0..4513831545b9a 100644 --- a/Objects/enumobject.c +++ b/Objects/enumobject.c @@ -1,6 +1,7 @@ /* enumerate object */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_long.h" // _PyLong_GetOne() #include "pycore_object.h" // _PyObject_GC_TRACK() diff --git a/Objects/fileobject.c b/Objects/fileobject.c index 5cb017e8e206e..dc600c6d09a48 100644 --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -2,7 +2,8 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pycore_runtime.h" // _PyRuntime +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_runtime.h" // _PyRuntime #if defined(HAVE_GETC_UNLOCKED) && !defined(_Py_MEMORY_SANITIZER) /* clang MemorySanitizer doesn't yet understand getc_unlocked. */ diff --git a/Objects/genobject.c b/Objects/genobject.c index 8fa04e8d68413..f91f367f9a687 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1,14 +1,15 @@ /* Generator object implementation */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _PyEval_EvalFrame() -#include "pycore_object.h" +#include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_pyerrors.h" // _PyErr_ClearExcState() #include "pycore_pystate.h" // _PyThreadState_GET() -#include "frameobject.h" -#include "pycore_frame.h" +#include "pycore_frame.h" // InterpreterFrame +#include "frameobject.h" // PyFrameObject #include "structmember.h" // PyMemberDef -#include "opcode.h" +#include "opcode.h" // YIELD_FROM static PyObject *gen_close(PyGenObject *, PyObject *); static PyObject *async_gen_asend_new(PyAsyncGenObject *, PyObject *); diff --git a/Objects/iterobject.c b/Objects/iterobject.c index acfc539fae0b7..5db6bc10fb3ff 100644 --- a/Objects/iterobject.c +++ b/Objects/iterobject.c @@ -1,7 +1,8 @@ /* Iterator objects */ #include "Python.h" -#include "pycore_object.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_object.h" // _PyObject_GC_TRACK() typedef struct { PyObject_HEAD diff --git a/Objects/moduleobject.c b/Objects/moduleobject.c index f008995f32f03..1d649a7932098 100644 --- a/Objects/moduleobject.c +++ b/Objects/moduleobject.c @@ -2,6 +2,7 @@ /* Module object implementation */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_interp.h" // PyInterpreterState.importlib #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_moduleobject.h" // _PyModule_GetDef() diff --git a/Objects/object.c b/Objects/object.c index d269274c74ec0..9d48346d6d227 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -2,18 +2,19 @@ /* Generic object operations; and implementation of None */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _Py_EnterRecursiveCall() #include "pycore_context.h" -#include "pycore_initconfig.h" -#include "pycore_object.h" -#include "pycore_pyerrors.h" -#include "pycore_pylifecycle.h" +#include "pycore_initconfig.h" // _PyStatus_EXCEPTION() +#include "pycore_object.h" // _PyType_CheckConsistency() +#include "pycore_pyerrors.h" // _PyErr_Occurred() +#include "pycore_pylifecycle.h" // _PyTypes_InitSlotDefs() #include "pycore_pymem.h" // _PyMem_IsPtrFreed() #include "pycore_pystate.h" // _PyThreadState_GET() #include "pycore_symtable.h" // PySTEntry_Type #include "pycore_unionobject.h" // _PyUnion_Type -#include "frameobject.h" -#include "interpreteridobject.h" +#include "frameobject.h" // PyFrame_Type +#include "interpreteridobject.h" // _PyInterpreterID_Type #ifdef Py_LIMITED_API // Prevent recursive call _Py_IncRef() <=> Py_INCREF() diff --git a/Objects/odictobject.c b/Objects/odictobject.c index 1462a3b397752..9af45c685ab3b 100644 --- a/Objects/odictobject.c +++ b/Objects/odictobject.c @@ -465,10 +465,10 @@ Potential Optimizations */ #include "Python.h" -#include "pycore_object.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_object.h" // _PyObject_GC_UNTRACK() +#include "pycore_dict.h" // _Py_dict_lookup() #include // offsetof() -#include "pycore_dict.h" -#include #include "clinic/odictobject.c.h" diff --git a/Objects/typeobject.c b/Objects/typeobject.c index a73a696a1ec88..0f56e59bc5185 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -4,14 +4,14 @@ #include "pycore_call.h" #include "pycore_code.h" // CO_FAST_FREE #include "pycore_compile.h" // _Py_Mangle() -#include "pycore_initconfig.h" +#include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_moduleobject.h" // _PyModule_GetDef() -#include "pycore_object.h" -#include "pycore_pyerrors.h" +#include "pycore_object.h" // _PyType_HasFeature() +#include "pycore_pyerrors.h" // _PyErr_Occurred() #include "pycore_pystate.h" // _PyThreadState_GET() #include "pycore_unionobject.h" // _Py_union_type_or -#include "frameobject.h" -#include "pycore_frame.h" +#include "frameobject.h" // PyFrameObject +#include "pycore_frame.h" // InterpreterFrame #include "opcode.h" // MAKE_CELL #include "structmember.h" // PyMemberDef diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 0e3caf3086ad9..c7a014de42d8f 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -3,6 +3,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include #include diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index f91bb9ffdbcf3..d07ba38b698c6 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -3,6 +3,7 @@ #include "Python.h" #include #include "pycore_ast.h" // _PyAST_Validate() +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_compile.h" // _PyAST_Compile() #include "pycore_object.h" // _Py_AddToAllObjects() #include "pycore_pyerrors.h" // _PyErr_NoMemory() diff --git a/Python/ceval.c b/Python/ceval.c index e9a9c1dccd82c..2d617a6364880 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -13,11 +13,11 @@ #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_call.h" // _PyObject_FastCallDictTstate() #include "pycore_ceval.h" // _PyEval_SignalAsyncExc() -#include "pycore_code.h" +#include "pycore_code.h" // saturating_increment() #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_GC_TRACK() -#include "pycore_moduleobject.h" +#include "pycore_moduleobject.h" // PyModuleObject #include "pycore_pyerrors.h" // _PyErr_Fetch() #include "pycore_pylifecycle.h" // _PyErr_Print() #include "pycore_pymem.h" // _PyMem_IsPtrFreed() diff --git a/Python/codecs.c b/Python/codecs.c index 9ee566bd0f761..b7c8db7e8b618 100644 --- a/Python/codecs.c +++ b/Python/codecs.c @@ -9,6 +9,7 @@ Copyright (c) Corporation for National Research Initiatives. ------------------------------------------------------------------------ */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_interp.h" // PyInterpreterState.codec_search_path #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_ucnhash.h" // _PyUnicode_Name_CAPI diff --git a/Python/errors.c b/Python/errors.c index 36a529e65a170..f072c21496592 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -2,11 +2,12 @@ /* Error handling */ #include "Python.h" -#include "pycore_initconfig.h" -#include "pycore_pyerrors.h" -#include "pycore_pystate.h" // _PyThreadState_GET() -#include "pycore_sysmodule.h" -#include "pycore_traceback.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_initconfig.h" // _PyStatus_ERR() +#include "pycore_pyerrors.h" // _PyErr_Format() +#include "pycore_pystate.h" // _PyThreadState_GET() +#include "pycore_sysmodule.h" // _PySys_Audit() +#include "pycore_traceback.h" // _PyTraceBack_FromFrame() #ifndef __STDC__ #ifndef MS_WINDOWS diff --git a/Python/marshal.c b/Python/marshal.c index 530c8d0f23857..c8a48a55dee0f 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -9,11 +9,12 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_code.h" // _PyCode_New() +#include "pycore_hashtable.h" // _Py_hashtable_t #include "longintrepr.h" #include "code.h" -#include "marshal.h" -#include "pycore_hashtable.h" -#include "pycore_code.h" // _PyCode_New() +#include "marshal.h" // Py_MARSHAL_VERSION /*[clinic input] module marshal diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 0154af0b76fba..ae3cbf1954e09 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -15,21 +15,22 @@ Data members: */ #include "Python.h" +#include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _Py_RecursionLimitLowerWaterMark() +#include "pycore_code.h" // _Py_QuickenedCount +#include "pycore_frame.h" // InterpreterFrame #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() #include "pycore_object.h" // _PyObject_IS_GC() -#include "pycore_code.h" // _Py_QuickenedCount #include "pycore_pathconfig.h" // _PyPathConfig_ComputeSysPath0() #include "pycore_pyerrors.h" // _PyErr_Fetch() #include "pycore_pylifecycle.h" // _PyErr_WriteUnraisableDefaultHook() #include "pycore_pymem.h" // _PyMem_SetDefaultAllocator() #include "pycore_pystate.h" // _PyThreadState_GET() -#include "pycore_tuple.h" // _PyTuple_FromArray() #include "pycore_structseq.h" // PyStructSequence_InitType() +#include "pycore_tuple.h" // _PyTuple_FromArray() #include "code.h" #include "frameobject.h" // PyFrame_GetBack() -#include "pycore_frame.h" #include "pydtrace.h" #include "osdefs.h" // DELIM #include "stdlib_module_names.h" // _Py_stdlib_module_names diff --git a/setup.py b/setup.py index 6122430cd0431..c6290eebed30f 100644 --- a/setup.py +++ b/setup.py @@ -942,7 +942,8 @@ def detect_simple_extensions(self): extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) # profiler (_lsprof is for cProfile.py) - self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c'])) + self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c'], + extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) # static Unicode character database self.add(Extension('unicodedata', ['unicodedata.c'], depends=['unicodedata_db.h', 'unicodename_db.h'], From webhook-mailer at python.org Tue Oct 12 03:17:58 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 07:17:58 -0000 Subject: [Python-checkins] bpo-42253: Update xml.dom.minidom.rst (GH-23126) (GH-28874) Message-ID: https://github.com/python/cpython/commit/3d7009e88e0ae516b10d8d3d402cc66e86fb631e commit: 3d7009e88e0ae516b10d8d3d402cc66e86fb631e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-12T10:17:53+03:00 summary: bpo-42253: Update xml.dom.minidom.rst (GH-23126) (GH-28874) Document that the "standalone" parameter was added in Python 3.9. Co-authored-by: Serhiy Storchaka (cherry picked from commit c7e81fcf9548ab6a0a4828d6f2db9ece9d204826) Co-authored-by: Jens Diemer files: M Doc/library/xml.dom.minidom.rst diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index d3a5f872204a3..20984b98b1778 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -156,6 +156,9 @@ module documentation. This section lists the differences between the API and The :meth:`writexml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toxml(encoding=None, standalone=None) Return a string or byte string containing the XML represented by @@ -174,6 +177,9 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None, \ standalone=None) @@ -190,6 +196,8 @@ module documentation. This section lists the differences between the API and The :meth:`toprettyxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. .. _dom-example: From webhook-mailer at python.org Tue Oct 12 03:18:20 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 07:18:20 -0000 Subject: [Python-checkins] bpo-42253: Update xml.dom.minidom.rst (GH-23126) (GH-28875) Message-ID: https://github.com/python/cpython/commit/23528905d48ddfe74a75af2152e96c04b77bf314 commit: 23528905d48ddfe74a75af2152e96c04b77bf314 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-12T10:18:15+03:00 summary: bpo-42253: Update xml.dom.minidom.rst (GH-23126) (GH-28875) Document that the "standalone" parameter was added in Python 3.9. Co-authored-by: Serhiy Storchaka (cherry picked from commit c7e81fcf9548ab6a0a4828d6f2db9ece9d204826) Co-authored-by: Jens Diemer files: M Doc/library/xml.dom.minidom.rst diff --git a/Doc/library/xml.dom.minidom.rst b/Doc/library/xml.dom.minidom.rst index d3a5f872204a3..20984b98b1778 100644 --- a/Doc/library/xml.dom.minidom.rst +++ b/Doc/library/xml.dom.minidom.rst @@ -156,6 +156,9 @@ module documentation. This section lists the differences between the API and The :meth:`writexml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toxml(encoding=None, standalone=None) Return a string or byte string containing the XML represented by @@ -174,6 +177,9 @@ module documentation. This section lists the differences between the API and The :meth:`toxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. + .. method:: Node.toprettyxml(indent="\\t", newl="\\n", encoding=None, \ standalone=None) @@ -190,6 +196,8 @@ module documentation. This section lists the differences between the API and The :meth:`toprettyxml` method now preserves the attribute order specified by the user. + .. versionchanged:: 3.9 + The *standalone* parameter was added. .. _dom-example: From webhook-mailer at python.org Tue Oct 12 03:18:47 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 07:18:47 -0000 Subject: [Python-checkins] bpo-45401: Fix a resource warning in test_logging (GH-28864) (GH-28873) Message-ID: https://github.com/python/cpython/commit/faa87f7f3b60f79b9018aaef0efa5e00d82b817b commit: faa87f7f3b60f79b9018aaef0efa5e00d82b817b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-12T10:18:43+03:00 summary: bpo-45401: Fix a resource warning in test_logging (GH-28864) (GH-28873) (cherry picked from commit 15188b115a2da815556053372c912a81a74be43b) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_logging.py diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 8202879df067c..876c91f64eb7c 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5174,6 +5174,7 @@ def test_should_not_rollover(self): time.sleep(1.1) # a little over a second ... r = logging.makeLogRecord({'msg': 'testing - device file'}) self.assertFalse(fh.shouldRollover(r)) + fh.close() # other test methods added below def test_rollover(self): From webhook-mailer at python.org Tue Oct 12 03:19:14 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 07:19:14 -0000 Subject: [Python-checkins] bpo-45401: Fix a resource warning in test_logging (GH-28864) (GH-28872) Message-ID: https://github.com/python/cpython/commit/47a50fe16f9f074daaaa3e6aa85e76502955ed40 commit: 47a50fe16f9f074daaaa3e6aa85e76502955ed40 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-12T10:19:09+03:00 summary: bpo-45401: Fix a resource warning in test_logging (GH-28864) (GH-28872) (cherry picked from commit 15188b115a2da815556053372c912a81a74be43b) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_logging.py diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 1a5f8b60f5ccb..93f6e5f85a0a3 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5298,6 +5298,7 @@ def test_should_not_rollover(self): time.sleep(1.1) # a little over a second ... r = logging.makeLogRecord({'msg': 'testing - device file'}) self.assertFalse(fh.shouldRollover(r)) + fh.close() # other test methods added below def test_rollover(self): From webhook-mailer at python.org Tue Oct 12 03:20:09 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 07:20:09 -0000 Subject: [Python-checkins] Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) Message-ID: https://github.com/python/cpython/commit/f79f3b41c8c1360d4e0ae884a52d0a486974ca53 commit: f79f3b41c8c1360d4e0ae884a52d0a486974ca53 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-12T10:20:04+03:00 summary: Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 3d9cd1ac8673c..6d2554741f982 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -207,7 +207,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) /* don't allow legacy init for non-ASCII module names */ PyErr_Format( PyExc_SystemError, - "initialization of * did not return PyModuleDef", + "initialization of %s did not return PyModuleDef", name_buf); goto error; } From webhook-mailer at python.org Tue Oct 12 05:29:33 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 09:29:33 -0000 Subject: [Python-checkins] Slight correct grammar (GH-28860) Message-ID: https://github.com/python/cpython/commit/1b11582f0e00f08833da6adc44030a8b1961989a commit: 1b11582f0e00f08833da6adc44030a8b1961989a branch: main author: nobodyatandnothing <91722596+nobodyatandnothing at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-12T12:29:29+03:00 summary: Slight correct grammar (GH-28860) files: M Doc/library/ipaddress.rst diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 74d922d29db69..9c2dff5570327 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -682,7 +682,7 @@ dictionaries. Note that currently expanded netmasks are not supported. That means ``2001:db00::0/24`` is a valid argument while ``2001:db00::0/ffff:ff00::`` - not. + is not. 2. An integer that fits into 128 bits. This is equivalent to a single-address network, with the network address being *address* and From webhook-mailer at python.org Tue Oct 12 06:36:23 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Tue, 12 Oct 2021 10:36:23 -0000 Subject: [Python-checkins] bpo-45441: Update some moved URLs in documentation (GH-28861) Message-ID: https://github.com/python/cpython/commit/b37dc9b3bc9575adc039c6093c643b7ae5e917e1 commit: b37dc9b3bc9575adc039c6093c643b7ae5e917e1 branch: main author: 180909 committer: serhiy-storchaka date: 2021-10-12T13:36:14+03:00 summary: bpo-45441: Update some moved URLs in documentation (GH-28861) files: M Doc/library/hashlib.rst M Doc/library/html.entities.rst M Doc/library/http.cookiejar.rst M Doc/library/json.rst diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 77b35fd1d766c..0c3bd7b5ac2c9 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -500,7 +500,7 @@ Keyed hashing Keyed hashing can be used for authentication as a faster and simpler replacement for `Hash-based message authentication code -`_ (HMAC). +`_ (HMAC). BLAKE2 can be securely used in prefix-MAC mode thanks to the indifferentiability property inherited from BLAKE. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index 067e1b1e5adb3..7d836fe738024 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -44,4 +44,4 @@ This module defines four dictionaries, :data:`html5`, .. rubric:: Footnotes -.. [#] See https://www.w3.org/TR/html5/syntax.html#named-character-references +.. [#] See https://html.spec.whatwg.org/multipage/syntax.html#named-character-references diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 6234e65629ea0..3d59665be4f25 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -122,7 +122,7 @@ The following classes are provided: :mod:`http.cookiejar` and :mod:`http.cookies` modules do not depend on each other. - https://curl.haxx.se/rfc/cookie_spec.html + https://curl.se/rfc/cookie_spec.html The specification of the original Netscape cookie protocol. Though this is still the dominant protocol, the 'Netscape cookie protocol' implemented by all the major browsers (and :mod:`http.cookiejar`) only bears a passing resemblance to diff --git a/Doc/library/json.rst b/Doc/library/json.rst index c8184da80fe43..6fa89f578a2cf 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -11,9 +11,9 @@ -------------- -`JSON (JavaScript Object Notation) `_, specified by +`JSON (JavaScript Object Notation) `_, specified by :rfc:`7159` (which obsoletes :rfc:`4627`) and by -`ECMA-404 `_, +`ECMA-404 `_, is a lightweight data interchange format inspired by `JavaScript `_ object literal syntax (although it is not a strict subset of JavaScript [#rfc-errata]_ ). @@ -544,7 +544,7 @@ Standard Compliance and Interoperability ---------------------------------------- The JSON format is specified by :rfc:`7159` and by -`ECMA-404 `_. +`ECMA-404 `_. This section details this module's level of compliance with the RFC. For simplicity, :class:`JSONEncoder` and :class:`JSONDecoder` subclasses, and parameters other than those explicitly mentioned, are not considered. From webhook-mailer at python.org Tue Oct 12 07:39:20 2021 From: webhook-mailer at python.org (encukou) Date: Tue, 12 Oct 2021 11:39:20 -0000 Subject: [Python-checkins] bpo-44991: Normalise function and collation callback naming (GH-28209) Message-ID: https://github.com/python/cpython/commit/cfb1df3b71501a80ed57739181ec2ed30012c491 commit: cfb1df3b71501a80ed57739181ec2ed30012c491 branch: main author: Erlend Egeberg Aasland committer: encukou date: 2021-10-12T13:38:49+02:00 summary: bpo-44991: Normalise function and collation callback naming (GH-28209) files: M Modules/_sqlite/connection.c diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index b4badb111a3c7..d6d1fa8bf2876 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -667,7 +667,7 @@ set_sqlite_error(sqlite3_context *context, const char *msg) } static void -_pysqlite_func_callback(sqlite3_context *context, int argc, sqlite3_value **argv) +func_callback(sqlite3_context *context, int argc, sqlite3_value **argv) { PyGILState_STATE threadstate = PyGILState_Ensure(); @@ -908,7 +908,7 @@ pysqlite_connection_create_function_impl(pysqlite_Connection *self, return NULL; } rc = sqlite3_create_function_v2(self->db, name, narg, flags, ctx, - _pysqlite_func_callback, + func_callback, NULL, NULL, &destructor_callback); // will decref func @@ -1501,10 +1501,8 @@ pysqlite_connection_executescript(pysqlite_Connection *self, /* ------------------------- COLLATION CODE ------------------------ */ static int -pysqlite_collation_callback( - void* context, - int text1_length, const void* text1_data, - int text2_length, const void* text2_data) +collation_callback(void *context, int text1_length, const void *text1_data, + int text2_length, const void *text2_data) { PyGILState_STATE gilstate = PyGILState_Ensure(); @@ -1778,7 +1776,7 @@ pysqlite_connection_create_collation_impl(pysqlite_Connection *self, return NULL; } rc = sqlite3_create_collation_v2(self->db, name, flags, ctx, - &pysqlite_collation_callback, + &collation_callback, &destructor_callback); } From webhook-mailer at python.org Tue Oct 12 10:20:51 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 12 Oct 2021 14:20:51 -0000 Subject: [Python-checkins] bpo-45441: Update some moved URLs in documentation (GH-28861) Message-ID: https://github.com/python/cpython/commit/32866dd471709dd7f3bd63ce762dec72198483d2 commit: 32866dd471709dd7f3bd63ce762dec72198483d2 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T07:20:43-07:00 summary: bpo-45441: Update some moved URLs in documentation (GH-28861) (cherry picked from commit b37dc9b3bc9575adc039c6093c643b7ae5e917e1) Co-authored-by: 180909 files: M Doc/library/hashlib.rst M Doc/library/html.entities.rst M Doc/library/http.cookiejar.rst M Doc/library/json.rst diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index 77b35fd1d766c..0c3bd7b5ac2c9 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -500,7 +500,7 @@ Keyed hashing Keyed hashing can be used for authentication as a faster and simpler replacement for `Hash-based message authentication code -`_ (HMAC). +`_ (HMAC). BLAKE2 can be securely used in prefix-MAC mode thanks to the indifferentiability property inherited from BLAKE. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index 067e1b1e5adb3..7d836fe738024 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -44,4 +44,4 @@ This module defines four dictionaries, :data:`html5`, .. rubric:: Footnotes -.. [#] See https://www.w3.org/TR/html5/syntax.html#named-character-references +.. [#] See https://html.spec.whatwg.org/multipage/syntax.html#named-character-references diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 6234e65629ea0..3d59665be4f25 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -122,7 +122,7 @@ The following classes are provided: :mod:`http.cookiejar` and :mod:`http.cookies` modules do not depend on each other. - https://curl.haxx.se/rfc/cookie_spec.html + https://curl.se/rfc/cookie_spec.html The specification of the original Netscape cookie protocol. Though this is still the dominant protocol, the 'Netscape cookie protocol' implemented by all the major browsers (and :mod:`http.cookiejar`) only bears a passing resemblance to diff --git a/Doc/library/json.rst b/Doc/library/json.rst index c8184da80fe43..6fa89f578a2cf 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -11,9 +11,9 @@ -------------- -`JSON (JavaScript Object Notation) `_, specified by +`JSON (JavaScript Object Notation) `_, specified by :rfc:`7159` (which obsoletes :rfc:`4627`) and by -`ECMA-404 `_, +`ECMA-404 `_, is a lightweight data interchange format inspired by `JavaScript `_ object literal syntax (although it is not a strict subset of JavaScript [#rfc-errata]_ ). @@ -544,7 +544,7 @@ Standard Compliance and Interoperability ---------------------------------------- The JSON format is specified by :rfc:`7159` and by -`ECMA-404 `_. +`ECMA-404 `_. This section details this module's level of compliance with the RFC. For simplicity, :class:`JSONEncoder` and :class:`JSONDecoder` subclasses, and parameters other than those explicitly mentioned, are not considered. From webhook-mailer at python.org Tue Oct 12 10:20:55 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 12 Oct 2021 14:20:55 -0000 Subject: [Python-checkins] bpo-45441: Update some moved URLs in documentation (GH-28861) Message-ID: https://github.com/python/cpython/commit/38e3ada75a7b5d911d81e8e4b70a99e181093866 commit: 38e3ada75a7b5d911d81e8e4b70a99e181093866 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T07:20:50-07:00 summary: bpo-45441: Update some moved URLs in documentation (GH-28861) (cherry picked from commit b37dc9b3bc9575adc039c6093c643b7ae5e917e1) Co-authored-by: 180909 files: M Doc/library/hashlib.rst M Doc/library/html.entities.rst M Doc/library/http.cookiejar.rst M Doc/library/json.rst diff --git a/Doc/library/hashlib.rst b/Doc/library/hashlib.rst index f3e942b8af222..58ccbbedb2c63 100644 --- a/Doc/library/hashlib.rst +++ b/Doc/library/hashlib.rst @@ -496,7 +496,7 @@ Keyed hashing Keyed hashing can be used for authentication as a faster and simpler replacement for `Hash-based message authentication code -`_ (HMAC). +`_ (HMAC). BLAKE2 can be securely used in prefix-MAC mode thanks to the indifferentiability property inherited from BLAKE. diff --git a/Doc/library/html.entities.rst b/Doc/library/html.entities.rst index 067e1b1e5adb3..7d836fe738024 100644 --- a/Doc/library/html.entities.rst +++ b/Doc/library/html.entities.rst @@ -44,4 +44,4 @@ This module defines four dictionaries, :data:`html5`, .. rubric:: Footnotes -.. [#] See https://www.w3.org/TR/html5/syntax.html#named-character-references +.. [#] See https://html.spec.whatwg.org/multipage/syntax.html#named-character-references diff --git a/Doc/library/http.cookiejar.rst b/Doc/library/http.cookiejar.rst index 9ac5d52a2ab09..fb113d89e9753 100644 --- a/Doc/library/http.cookiejar.rst +++ b/Doc/library/http.cookiejar.rst @@ -122,7 +122,7 @@ The following classes are provided: :mod:`http.cookiejar` and :mod:`http.cookies` modules do not depend on each other. - https://curl.haxx.se/rfc/cookie_spec.html + https://curl.se/rfc/cookie_spec.html The specification of the original Netscape cookie protocol. Though this is still the dominant protocol, the 'Netscape cookie protocol' implemented by all the major browsers (and :mod:`http.cookiejar`) only bears a passing resemblance to diff --git a/Doc/library/json.rst b/Doc/library/json.rst index c8184da80fe43..6fa89f578a2cf 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -11,9 +11,9 @@ -------------- -`JSON (JavaScript Object Notation) `_, specified by +`JSON (JavaScript Object Notation) `_, specified by :rfc:`7159` (which obsoletes :rfc:`4627`) and by -`ECMA-404 `_, +`ECMA-404 `_, is a lightweight data interchange format inspired by `JavaScript `_ object literal syntax (although it is not a strict subset of JavaScript [#rfc-errata]_ ). @@ -544,7 +544,7 @@ Standard Compliance and Interoperability ---------------------------------------- The JSON format is specified by :rfc:`7159` and by -`ECMA-404 `_. +`ECMA-404 `_. This section details this module's level of compliance with the RFC. For simplicity, :class:`JSONEncoder` and :class:`JSONDecoder` subclasses, and parameters other than those explicitly mentioned, are not considered. From webhook-mailer at python.org Tue Oct 12 13:01:15 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 17:01:15 -0000 Subject: [Python-checkins] Slight correct grammar (GH-28860) Message-ID: https://github.com/python/cpython/commit/035ad8cf988b31a742ec006183a0ea673876d659 commit: 035ad8cf988b31a742ec006183a0ea673876d659 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-12T10:01:11-07:00 summary: Slight correct grammar (GH-28860) (cherry picked from commit 1b11582f0e00f08833da6adc44030a8b1961989a) Co-authored-by: nobodyatandnothing <91722596+nobodyatandnothing at users.noreply.github.com> files: M Doc/library/ipaddress.rst diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 74d922d29db69..9c2dff5570327 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -682,7 +682,7 @@ dictionaries. Note that currently expanded netmasks are not supported. That means ``2001:db00::0/24`` is a valid argument while ``2001:db00::0/ffff:ff00::`` - not. + is not. 2. An integer that fits into 128 bits. This is equivalent to a single-address network, with the network address being *address* and From webhook-mailer at python.org Tue Oct 12 13:01:31 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 17:01:31 -0000 Subject: [Python-checkins] Slight correct grammar (GH-28860) Message-ID: https://github.com/python/cpython/commit/0cd5bf919430755b4e2c08c0f2e42f74ae2f00ad commit: 0cd5bf919430755b4e2c08c0f2e42f74ae2f00ad branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-12T10:01:26-07:00 summary: Slight correct grammar (GH-28860) (cherry picked from commit 1b11582f0e00f08833da6adc44030a8b1961989a) Co-authored-by: nobodyatandnothing <91722596+nobodyatandnothing at users.noreply.github.com> files: M Doc/library/ipaddress.rst diff --git a/Doc/library/ipaddress.rst b/Doc/library/ipaddress.rst index 74d922d29db69..9c2dff5570327 100644 --- a/Doc/library/ipaddress.rst +++ b/Doc/library/ipaddress.rst @@ -682,7 +682,7 @@ dictionaries. Note that currently expanded netmasks are not supported. That means ``2001:db00::0/24`` is a valid argument while ``2001:db00::0/ffff:ff00::`` - not. + is not. 2. An integer that fits into 128 bits. This is equivalent to a single-address network, with the network address being *address* and From webhook-mailer at python.org Tue Oct 12 13:10:50 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 17:10:50 -0000 Subject: [Python-checkins] Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) Message-ID: https://github.com/python/cpython/commit/6d4d4ec59f3e8cf633af882b386959ecbe1e0c54 commit: 6d4d4ec59f3e8cf633af882b386959ecbe1e0c54 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-12T10:10:41-07:00 summary: Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) (cherry picked from commit f79f3b41c8c1360d4e0ae884a52d0a486974ca53) Co-authored-by: Serhiy Storchaka files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 7f600d4a7aaf8..134f6680c4484 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -207,7 +207,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) /* don't allow legacy init for non-ASCII module names */ PyErr_Format( PyExc_SystemError, - "initialization of * did not return PyModuleDef", + "initialization of %s did not return PyModuleDef", name_buf); goto error; } From webhook-mailer at python.org Tue Oct 12 13:11:03 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 17:11:03 -0000 Subject: [Python-checkins] Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) Message-ID: https://github.com/python/cpython/commit/854db7e82126701f2bf7a3ca0d78e3e26096aa75 commit: 854db7e82126701f2bf7a3ca0d78e3e26096aa75 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: Mariatta date: 2021-10-12T10:10:59-07:00 summary: Fix format string in _PyImport_LoadDynamicModuleWithSpec() (GH-28863) (cherry picked from commit f79f3b41c8c1360d4e0ae884a52d0a486974ca53) Co-authored-by: Serhiy Storchaka files: M Python/importdl.c diff --git a/Python/importdl.c b/Python/importdl.c index 3d9cd1ac8673c..6d2554741f982 100644 --- a/Python/importdl.c +++ b/Python/importdl.c @@ -207,7 +207,7 @@ _PyImport_LoadDynamicModuleWithSpec(PyObject *spec, FILE *fp) /* don't allow legacy init for non-ASCII module names */ PyErr_Format( PyExc_SystemError, - "initialization of * did not return PyModuleDef", + "initialization of %s did not return PyModuleDef", name_buf); goto error; } From webhook-mailer at python.org Tue Oct 12 13:12:27 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 17:12:27 -0000 Subject: [Python-checkins] bpo-45421: Remove dead code from html.parser (GH-28847) Message-ID: https://github.com/python/cpython/commit/562c0d7398b9f34ff63a1dc77113dad96a93ce4e commit: 562c0d7398b9f34ff63a1dc77113dad96a93ce4e branch: main author: Alberto Mardegan committer: Mariatta date: 2021-10-12T10:12:21-07:00 summary: bpo-45421: Remove dead code from html.parser (GH-28847) Support for HtmlParserError was removed back in 2014 with commit 73a4359eb0eb624c588c5d52083ea4944f9787ea, however this small block was missed. files: M Lib/html/parser.py diff --git a/Lib/html/parser.py b/Lib/html/parser.py index 58f6bb3b1e932..bef0f4fe4bf77 100644 --- a/Lib/html/parser.py +++ b/Lib/html/parser.py @@ -328,13 +328,6 @@ def parse_starttag(self, i): end = rawdata[k:endpos].strip() if end not in (">", "/>"): - lineno, offset = self.getpos() - if "\n" in self.__starttag_text: - lineno = lineno + self.__starttag_text.count("\n") - offset = len(self.__starttag_text) \ - - self.__starttag_text.rfind("\n") - else: - offset = offset + len(self.__starttag_text) self.handle_data(rawdata[i:endpos]) return endpos if end.endswith('/>'): From webhook-mailer at python.org Tue Oct 12 14:22:10 2021 From: webhook-mailer at python.org (Mariatta) Date: Tue, 12 Oct 2021 18:22:10 -0000 Subject: [Python-checkins] Fix spelling in Misc (GH-28858) Message-ID: https://github.com/python/cpython/commit/2d21612f0dd84bf6d0ce35bcfcc9f0e1a41c202d commit: 2d21612f0dd84bf6d0ce35bcfcc9f0e1a41c202d branch: main author: 180909 committer: Mariatta date: 2021-10-12T11:22:05-07:00 summary: Fix spelling in Misc (GH-28858) files: M Misc/HISTORY diff --git a/Misc/HISTORY b/Misc/HISTORY index 32d814c31bfd1..805acf4bebcad 100644 --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -1225,7 +1225,7 @@ Library - Issue #22396: On 32-bit AIX platform, don't expose os.posix_fadvise() nor os.posix_fallocate() because their prototypes in system headers are wrong. -- Issue #22517: When a io.BufferedRWPair object is deallocated, clear its +- Issue #22517: When an io.BufferedRWPair object is deallocated, clear its weakrefs. - Issue #22448: Improve canceled timer handles cleanup to prevent From webhook-mailer at python.org Tue Oct 12 19:52:30 2021 From: webhook-mailer at python.org (vstinner) Date: Tue, 12 Oct 2021 23:52:30 -0000 Subject: [Python-checkins] bpo-45410: Enhance libregrtest -W/--verbose3 option (GH-28908) Message-ID: https://github.com/python/cpython/commit/dbe213de7ef28712bbfdb9d94a33abb9c33ef0c2 commit: dbe213de7ef28712bbfdb9d94a33abb9c33ef0c2 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T01:52:22+02:00 summary: bpo-45410: Enhance libregrtest -W/--verbose3 option (GH-28908) libregrtest -W/--verbose3 now also replace sys.__stdout__, sys.__stderr__, and stdout and stderr file descriptors (fd 1 and fd 2). support.print_warning() messages are now logged in the expected order. The "./python -m test test_eintr -W" command no longer logs into stdout if the test pass. files: M Lib/test/libregrtest/runtest.py M Lib/test/libregrtest/utils.py M Lib/test/support/__init__.py diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index fe4693bad9ca6..31474a222ed09 100644 --- a/Lib/test/libregrtest/runtest.py +++ b/Lib/test/libregrtest/runtest.py @@ -1,3 +1,4 @@ +import contextlib import faulthandler import functools import gc @@ -5,6 +6,7 @@ import io import os import sys +import tempfile import time import traceback import unittest @@ -173,6 +175,63 @@ def get_abs_module(ns: Namespace, test_name: str) -> str: return 'test.' + test_name + at contextlib.contextmanager +def override_fd(fd, fd2): + fd2_copy = os.dup(fd2) + try: + os.dup2(fd, fd2) + yield + finally: + os.dup2(fd2_copy, fd2) + os.close(fd2_copy) + + +def get_stream_fd(stream): + if stream is None: + return None + try: + return stream.fileno() + except io.UnsupportedOperation: + return None + + + at contextlib.contextmanager +def capture_std_streams(): + """ + Redirect all standard streams to a temporary file: + + * stdout and stderr file descriptors (fd 1 and fd 2) + * sys.stdout, sys.__stdout__ + * sys.stderr, sys.__stderr__ + """ + try: + stderr_fd = sys.stderr.fileno() + except io.UnsupportedOperation: + stderr_fd = None + + # Use a temporary file to support fileno() operation + tmp_file = tempfile.TemporaryFile(mode='w+', + # line buffering + buffering=1, + encoding=sys.stderr.encoding, + errors=sys.stderr.errors) + with contextlib.ExitStack() as stack: + stack.enter_context(tmp_file) + + # Override stdout and stderr file descriptors + tmp_fd = tmp_file.fileno() + for stream in (sys.stdout, sys.stderr): + fd = get_stream_fd(stream) + if fd is not None: + stack.enter_context(override_fd(tmp_fd, fd)) + + # Override sys attributes + for name in ('stdout', 'stderr', '__stdout__', '__stderr__'): + stack.enter_context(support.swap_attr(sys, name, tmp_file)) + + yield tmp_file + + def _runtest(ns: Namespace, test_name: str) -> TestResult: # Handle faulthandler timeout, capture stdout+stderr, XML serialization # and measure time. @@ -193,21 +252,17 @@ def _runtest(ns: Namespace, test_name: str) -> TestResult: if output_on_failure: support.verbose = True - stream = io.StringIO() - orig_stdout = sys.stdout - orig_stderr = sys.stderr - try: - sys.stdout = stream - sys.stderr = stream + output = None + with capture_std_streams() as stream: result = _runtest_inner(ns, test_name, display_failure=False) if not isinstance(result, Passed): - output = stream.getvalue() - orig_stderr.write(output) - orig_stderr.flush() - finally: - sys.stdout = orig_stdout - sys.stderr = orig_stderr + stream.seek(0) + output = stream.read() + + if output is not None: + sys.stderr.write(output) + sys.stderr.flush() else: # Tell tests to be moderately quiet support.verbose = ns.verbose diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index c71467a51921d..256f9a4cb6eed 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -71,7 +71,7 @@ def print_warning(msg): def regrtest_unraisable_hook(unraisable): global orig_unraisablehook support.environment_altered = True - print_warning("Unraisable exception") + support.print_warning("Unraisable exception") old_stderr = sys.stderr try: support.flush_std_streams() @@ -94,7 +94,7 @@ def setup_unraisable_hook(): def regrtest_threading_excepthook(args): global orig_threading_excepthook support.environment_altered = True - print_warning(f"Uncaught thread exception: {args.exc_type.__name__}") + support.print_warning(f"Uncaught thread exception: {args.exc_type.__name__}") old_stderr = sys.stderr try: support.flush_std_streams() diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 0bbe813775ce5..45801dc317a6a 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1179,8 +1179,10 @@ def print_warning(msg): flush_std_streams() # bpo-39983: Print into sys.__stderr__ to display the warning even # when sys.stderr is captured temporarily by a test + stream = sys.__stderr__ for line in msg.splitlines(): - print(f"Warning -- {line}", file=sys.__stderr__, flush=True) + print(f"Warning -- {line}", file=stream) + stream.flush() # Flag used by saved_test_environment of test.libregrtest.save_env, From webhook-mailer at python.org Tue Oct 12 20:10:34 2021 From: webhook-mailer at python.org (ned-deily) Date: Wed, 13 Oct 2021 00:10:34 -0000 Subject: [Python-checkins] bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (#28845) Message-ID: https://github.com/python/cpython/commit/9c4766772cda67648184f8ddba546a5fc0167f91 commit: 9c4766772cda67648184f8ddba546a5fc0167f91 branch: main author: David Bohman committer: ned-deily date: 2021-10-12T20:10:26-04:00 summary: bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (#28845) Change the configure logic to function properly on macOS when the compiler outputs a platform triplet for option --print-multiarch. Co-authored-by: Ned Deily files: A Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst new file mode 100644 index 0000000000000..a2dc5bcc32217 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst @@ -0,0 +1,2 @@ +Prevent ``internal configure error`` when running ``configure`` +with recent versions of non-Apple clang. Patch by David Bohman. diff --git a/configure b/configure index 70f28b0c7064e..b2d1a64206cd4 100755 --- a/configure +++ b/configure @@ -5227,9 +5227,6 @@ $as_echo "$as_me: fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) - - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for the platform triplet based on compiler characteristics" >&5 $as_echo_n "checking for the platform triplet based on compiler characteristics... " >&6; } cat >> conftest.c <&6; } fi rm -f conftest.c conftest.out +if test x$PLATFORM_TRIPLET != xdarwin; then + MULTIARCH=$($CC --print-multiarch 2>/dev/null) +fi + + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then as_fn_error $? "internal configure error for the platform triplet, please file a bug report" "$LINENO" 5 diff --git a/configure.ac b/configure.ac index afdc68363ceaf..a6c6d1c49596a 100644 --- a/configure.ac +++ b/configure.ac @@ -717,9 +717,6 @@ then fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) -AC_SUBST(MULTIARCH) - AC_MSG_CHECKING([for the platform triplet based on compiler characteristics]) cat >> conftest.c </dev/null) +fi +AC_SUBST(MULTIARCH) + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report]) From webhook-mailer at python.org Tue Oct 12 20:31:40 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 13 Oct 2021 00:31:40 -0000 Subject: [Python-checkins] [3.10] bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (GH-28845) (GH-28911) Message-ID: https://github.com/python/cpython/commit/edae3e2ac73151217b4b4e096dd9f17cc0a50c11 commit: edae3e2ac73151217b4b4e096dd9f17cc0a50c11 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T17:31:32-07:00 summary: [3.10] bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (GH-28845) (GH-28911) Change the configure logic to function properly on macOS when the compiler outputs a platform triplet for option --print-multiarch. Co-authored-by: Ned Deily (cherry picked from commit 9c4766772cda67648184f8ddba546a5fc0167f91) Co-authored-by: David Bohman Automerge-Triggered-By: GH:ned-deily files: A Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst new file mode 100644 index 0000000000000..a2dc5bcc32217 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst @@ -0,0 +1,2 @@ +Prevent ``internal configure error`` when running ``configure`` +with recent versions of non-Apple clang. Patch by David Bohman. diff --git a/configure b/configure index 02d882ed39d68..a6e0f823b453a 100755 --- a/configure +++ b/configure @@ -5226,9 +5226,6 @@ $as_echo "$as_me: fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) - - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for the platform triplet based on compiler characteristics" >&5 $as_echo_n "checking for the platform triplet based on compiler characteristics... " >&6; } cat >> conftest.c <&6; } fi rm -f conftest.c conftest.out +if test x$PLATFORM_TRIPLET != xdarwin; then + MULTIARCH=$($CC --print-multiarch 2>/dev/null) +fi + + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then as_fn_error $? "internal configure error for the platform triplet, please file a bug report" "$LINENO" 5 diff --git a/configure.ac b/configure.ac index 41a3679d3dc55..c4abac6701152 100644 --- a/configure.ac +++ b/configure.ac @@ -717,9 +717,6 @@ then fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) -AC_SUBST(MULTIARCH) - AC_MSG_CHECKING([for the platform triplet based on compiler characteristics]) cat >> conftest.c </dev/null) +fi +AC_SUBST(MULTIARCH) + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report]) From webhook-mailer at python.org Tue Oct 12 20:36:07 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 13 Oct 2021 00:36:07 -0000 Subject: [Python-checkins] [3.9] bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (GH-28845) (GH-28910) Message-ID: https://github.com/python/cpython/commit/9901d153c201d852d27dc9d3074e283c26468f6d commit: 9901d153c201d852d27dc9d3074e283c26468f6d branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T17:35:59-07:00 summary: [3.9] bpo-45405: Prevent ``internal configure error`` when running ``configure`` with recent versions of non-Apple clang. (GH-28845) (GH-28910) Change the configure logic to function properly on macOS when the compiler outputs a platform triplet for option --print-multiarch. Co-authored-by: Ned Deily (cherry picked from commit 9c4766772cda67648184f8ddba546a5fc0167f91) Co-authored-by: David Bohman Automerge-Triggered-By: GH:ned-deily files: A Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst new file mode 100644 index 0000000000000..a2dc5bcc32217 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-11-16-27-38.bpo-45405.iSfdW5.rst @@ -0,0 +1,2 @@ +Prevent ``internal configure error`` when running ``configure`` +with recent versions of non-Apple clang. Patch by David Bohman. diff --git a/configure b/configure index 7cad0e2f98ba1..c0dfcd64c52a5 100755 --- a/configure +++ b/configure @@ -5196,9 +5196,6 @@ $as_echo "$as_me: fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) - - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for the platform triplet based on compiler characteristics" >&5 $as_echo_n "checking for the platform triplet based on compiler characteristics... " >&6; } cat >> conftest.c <&6; } fi rm -f conftest.c conftest.out +if test x$PLATFORM_TRIPLET != xdarwin; then + MULTIARCH=$($CC --print-multiarch 2>/dev/null) +fi + + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then as_fn_error $? "internal configure error for the platform triplet, please file a bug report" "$LINENO" 5 diff --git a/configure.ac b/configure.ac index be28e3a38f7ab..c9bf86b71a97f 100644 --- a/configure.ac +++ b/configure.ac @@ -727,9 +727,6 @@ then fi -MULTIARCH=$($CC --print-multiarch 2>/dev/null) -AC_SUBST(MULTIARCH) - AC_MSG_CHECKING([for the platform triplet based on compiler characteristics]) cat >> conftest.c </dev/null) +fi +AC_SUBST(MULTIARCH) + if test x$PLATFORM_TRIPLET != x && test x$MULTIARCH != x; then if test x$PLATFORM_TRIPLET != x$MULTIARCH; then AC_MSG_ERROR([internal configure error for the platform triplet, please file a bug report]) From webhook-mailer at python.org Tue Oct 12 21:39:58 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 01:39:58 -0000 Subject: [Python-checkins] bpo-45434: Convert Py_GETENV() macro to a function (GH-28912) Message-ID: https://github.com/python/cpython/commit/489176e4285314f9ea87b8bd91fe1d55d9af2c42 commit: 489176e4285314f9ea87b8bd91fe1d55d9af2c42 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T03:39:50+02:00 summary: bpo-45434: Convert Py_GETENV() macro to a function (GH-28912) Avoid calling directly getenv() in the header file. files: M Include/cpython/pydebug.h M Python/initconfig.c diff --git a/Include/cpython/pydebug.h b/Include/cpython/pydebug.h index 78bcb118be465..cab799f0b38e0 100644 --- a/Include/cpython/pydebug.h +++ b/Include/cpython/pydebug.h @@ -29,7 +29,7 @@ PyAPI_DATA(int) Py_LegacyWindowsStdioFlag; /* this is a wrapper around getenv() that pays attention to Py_IgnoreEnvironmentFlag. It should be used for getting variables like PYTHONPATH and PYTHONHOME from the environment */ -#define Py_GETENV(s) (Py_IgnoreEnvironmentFlag ? NULL : getenv(s)) +PyAPI_DATA(char*) Py_GETENV(const char *name); #ifdef __cplusplus } diff --git a/Python/initconfig.c b/Python/initconfig.c index 2e3cde83b2534..b91d280906cbe 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -249,6 +249,14 @@ _Py_GetGlobalVariablesAsDict(void) #undef SET_ITEM_STR } +char* +Py_GETENV(const char *name) +{ + if (Py_IgnoreEnvironmentFlag) { + return NULL; + } + return getenv(name); +} /* --- PyStatus ----------------------------------------------- */ From webhook-mailer at python.org Tue Oct 12 22:38:04 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 02:38:04 -0000 Subject: [Python-checkins] bpo-45434: bytearrayobject.h no longer includes (GH-28913) Message-ID: https://github.com/python/cpython/commit/c63623a0a6892ce8683dbf8c769793ea897e6ba8 commit: c63623a0a6892ce8683dbf8c769793ea897e6ba8 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T04:37:55+02:00 summary: bpo-45434: bytearrayobject.h no longer includes (GH-28913) bytearrayobject.h and _lzmamodule.c don't use va_list and so don't need to include . files: M Include/bytearrayobject.h M Include/bytesobject.h M Include/modsupport.h M Include/unicodeobject.h M Modules/_lzmamodule.c diff --git a/Include/bytearrayobject.h b/Include/bytearrayobject.h index 9e95433f0f26f..1a834474dde7c 100644 --- a/Include/bytearrayobject.h +++ b/Include/bytearrayobject.h @@ -6,8 +6,6 @@ extern "C" { #endif -#include - /* Type PyByteArrayObject represents a mutable array of bytes. * The Python API is that of a sequence; * the bytes are mapped to ints in [0, 256). diff --git a/Include/bytesobject.h b/Include/bytesobject.h index 39c241a2dcf5f..bcb1a5942c68f 100644 --- a/Include/bytesobject.h +++ b/Include/bytesobject.h @@ -7,7 +7,7 @@ extern "C" { #endif -#include +#include // va_list /* Type PyBytesObject represents a byte string. An extra zero byte is diff --git a/Include/modsupport.h b/Include/modsupport.h index 7d37b49942255..baf47f0038d08 100644 --- a/Include/modsupport.h +++ b/Include/modsupport.h @@ -7,7 +7,7 @@ extern "C" { /* Module support interface */ -#include +#include // va_list /* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier to mean Py_ssize_t */ diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h index b0ac086a6be23..c65b9228298d7 100644 --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1,7 +1,7 @@ #ifndef Py_UNICODEOBJECT_H #define Py_UNICODEOBJECT_H -#include +#include // va_list /* diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c index 915c0c918f644..a7156ec7ddf04 100644 --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -10,7 +10,6 @@ #include "Python.h" #include "structmember.h" // PyMemberDef -#include #include #include From webhook-mailer at python.org Tue Oct 12 23:24:43 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 03:24:43 -0000 Subject: [Python-checkins] bpo-45453: Fix test_embed.StdPrinterTests (GH-28916) Message-ID: https://github.com/python/cpython/commit/678433f25e0d08dad7edf72be8f0cf9420e4ed2c commit: 678433f25e0d08dad7edf72be8f0cf9420e4ed2c branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T05:24:33+02:00 summary: bpo-45453: Fix test_embed.StdPrinterTests (GH-28916) test_embed.StdPrinterTests now always use the file descriptor 1 for stdout, rather than using sys.__stdout__.fileno(). PyFile_NewStdPrinter() does crash if the argument is not 1 or 2. Fix also a few pyflakes warnings: remove unused import and variables. files: M Lib/test/test_embed.py diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index c748000f7005d..31c5c3e49dda1 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -13,7 +13,6 @@ import shutil import subprocess import sys -import sysconfig import tempfile import textwrap @@ -592,7 +591,6 @@ def _get_expected_config(self): def get_expected_config(self, expected_preconfig, expected, expected_pathconfig, env, api, modify_path_cb=None): - cls = self.__class__ configs = self._get_expected_config() pre_config = configs['pre_config'] @@ -1248,7 +1246,6 @@ def test_init_setpythonhome(self): self.fail(f"Unable to find home in {paths!r}") prefix = exec_prefix = home - ver = sys.version_info expected_paths = self.module_search_paths(prefix=home, exec_prefix=home) config = { @@ -1549,8 +1546,7 @@ class StdPrinterTests(EmbeddingTestsMixin, unittest.TestCase): # "Set up a preliminary stderr printer until we have enough # infrastructure for the io module in place." - def get_stdout_fd(self): - return sys.__stdout__.fileno() + STDOUT_FD = 1 def create_printer(self, fd): ctypes = import_helper.import_module('ctypes') @@ -1562,7 +1558,7 @@ def create_printer(self, fd): def test_write(self): message = "unicode:\xe9-\u20ac-\udc80!\n" - stdout_fd = self.get_stdout_fd() + stdout_fd = self.STDOUT_FD stdout_fd_copy = os.dup(stdout_fd) self.addCleanup(os.close, stdout_fd_copy) @@ -1583,7 +1579,7 @@ def test_write(self): self.assertEqual(data, message.encode('utf8', 'backslashreplace')) def test_methods(self): - fd = self.get_stdout_fd() + fd = self.STDOUT_FD printer = self.create_printer(fd) self.assertEqual(printer.fileno(), fd) self.assertEqual(printer.isatty(), os.isatty(fd)) @@ -1591,7 +1587,7 @@ def test_methods(self): printer.close() # noop def test_disallow_instantiation(self): - fd = self.get_stdout_fd() + fd = self.STDOUT_FD printer = self.create_printer(fd) support.check_disallow_instantiation(self, type(printer)) From webhook-mailer at python.org Wed Oct 13 01:15:07 2021 From: webhook-mailer at python.org (terryjreedy) Date: Wed, 13 Oct 2021 05:15:07 -0000 Subject: [Python-checkins] bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) Message-ID: https://github.com/python/cpython/commit/380c44087505d0d560f97e325028f27393551164 commit: 380c44087505d0d560f97e325028f27393551164 branch: main author: Terry Jan Reedy committer: terryjreedy date: 2021-10-13T01:14:58-04:00 summary: bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) To avoid error, add either space or parentheses. files: A Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index ef80808a1a4d5..12b70dbbe7302 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -836,6 +836,27 @@ ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to bite. +How do I get int literal attribute instead of SyntaxError? +---------------------------------------------------------- + +Trying to lookup an ``int`` literal attribute in the normal manner gives +a syntax error because the period is seen as a decimal point:: + + >>> 1.__class__ + File "", line 1 + 1.__class__ + ^ + SyntaxError: invalid decimal literal + +The solution is to separate the literal from the period +with either a space or parentheses. + + >>> 1 .__class__ + + >>> (1).__class__ + + + How do I convert a string to a number? -------------------------------------- diff --git a/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst new file mode 100644 index 0000000000000..44ae468d1bccf --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst @@ -0,0 +1,2 @@ +Add Programming FAQ entry explaining that int literal attribute access +requires either a space after or parentheses around the literal. From webhook-mailer at python.org Wed Oct 13 01:38:01 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 13 Oct 2021 05:38:01 -0000 Subject: [Python-checkins] bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) Message-ID: https://github.com/python/cpython/commit/47673c47db352916384e4f35fa520e48475e2601 commit: 47673c47db352916384e4f35fa520e48475e2601 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T22:37:51-07:00 summary: bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) To avoid error, add either space or parentheses. (cherry picked from commit 380c44087505d0d560f97e325028f27393551164) Co-authored-by: Terry Jan Reedy files: A Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index ef80808a1a4d5..12b70dbbe7302 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -836,6 +836,27 @@ ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to bite. +How do I get int literal attribute instead of SyntaxError? +---------------------------------------------------------- + +Trying to lookup an ``int`` literal attribute in the normal manner gives +a syntax error because the period is seen as a decimal point:: + + >>> 1.__class__ + File "", line 1 + 1.__class__ + ^ + SyntaxError: invalid decimal literal + +The solution is to separate the literal from the period +with either a space or parentheses. + + >>> 1 .__class__ + + >>> (1).__class__ + + + How do I convert a string to a number? -------------------------------------- diff --git a/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst new file mode 100644 index 0000000000000..44ae468d1bccf --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst @@ -0,0 +1,2 @@ +Add Programming FAQ entry explaining that int literal attribute access +requires either a space after or parentheses around the literal. From webhook-mailer at python.org Wed Oct 13 01:40:23 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 13 Oct 2021 05:40:23 -0000 Subject: [Python-checkins] bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) Message-ID: https://github.com/python/cpython/commit/cc90732d15b267feb4cb75ec4c448a3c66e6c520 commit: cc90732d15b267feb4cb75ec4c448a3c66e6c520 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-12T22:40:18-07:00 summary: bpo-20692: Add Programming FAQ entry for 1.__class__ error. (GH-28918) To avoid error, add either space or parentheses. (cherry picked from commit 380c44087505d0d560f97e325028f27393551164) Co-authored-by: Terry Jan Reedy files: A Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst M Doc/faq/programming.rst diff --git a/Doc/faq/programming.rst b/Doc/faq/programming.rst index 4e04b10b0dae6..5286bbbccf073 100644 --- a/Doc/faq/programming.rst +++ b/Doc/faq/programming.rst @@ -836,6 +836,27 @@ ago? ``-190 % 12 == 2`` is useful; ``-190 % 12 == -10`` is a bug waiting to bite. +How do I get int literal attribute instead of SyntaxError? +---------------------------------------------------------- + +Trying to lookup an ``int`` literal attribute in the normal manner gives +a syntax error because the period is seen as a decimal point:: + + >>> 1.__class__ + File "", line 1 + 1.__class__ + ^ + SyntaxError: invalid decimal literal + +The solution is to separate the literal from the period +with either a space or parentheses. + + >>> 1 .__class__ + + >>> (1).__class__ + + + How do I convert a string to a number? -------------------------------------- diff --git a/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst new file mode 100644 index 0000000000000..44ae468d1bccf --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-13-00-42-54.bpo-20692.K5rGtP.rst @@ -0,0 +1,2 @@ +Add Programming FAQ entry explaining that int literal attribute access +requires either a space after or parentheses around the literal. From webhook-mailer at python.org Wed Oct 13 06:24:57 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 10:24:57 -0000 Subject: [Python-checkins] bpo-45453: Fix test_embed.StdPrinterTests (GH-28916) (GH-28917) Message-ID: https://github.com/python/cpython/commit/7e74d99c532f3eb55a86d10d30c068b03f2ad025 commit: 7e74d99c532f3eb55a86d10d30c068b03f2ad025 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vstinner date: 2021-10-13T12:24:47+02:00 summary: bpo-45453: Fix test_embed.StdPrinterTests (GH-28916) (GH-28917) test_embed.StdPrinterTests now always use the file descriptor 1 for stdout, rather than using sys.__stdout__.fileno(). PyFile_NewStdPrinter() does crash if the argument is not 1 or 2. Fix also a few pyflakes warnings: remove unused import and variables. (cherry picked from commit 678433f25e0d08dad7edf72be8f0cf9420e4ed2c) Co-authored-by: Victor Stinner files: M Lib/test/test_embed.py diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index da9b555522f4f..503d4925c9402 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -588,7 +588,6 @@ def _get_expected_config(self): def get_expected_config(self, expected_preconfig, expected, expected_pathconfig, env, api, modify_path_cb=None): - cls = self.__class__ configs = self._get_expected_config() pre_config = configs['pre_config'] @@ -1235,7 +1234,6 @@ def test_init_setpythonhome(self): self.fail(f"Unable to find home in {paths!r}") prefix = exec_prefix = home - ver = sys.version_info expected_paths = self.module_search_paths(prefix=home, exec_prefix=home) config = { @@ -1505,8 +1503,7 @@ class StdPrinterTests(EmbeddingTestsMixin, unittest.TestCase): # "Set up a preliminary stderr printer until we have enough # infrastructure for the io module in place." - def get_stdout_fd(self): - return sys.__stdout__.fileno() + STDOUT_FD = 1 def create_printer(self, fd): ctypes = import_helper.import_module('ctypes') @@ -1518,7 +1515,7 @@ def create_printer(self, fd): def test_write(self): message = "unicode:\xe9-\u20ac-\udc80!\n" - stdout_fd = self.get_stdout_fd() + stdout_fd = self.STDOUT_FD stdout_fd_copy = os.dup(stdout_fd) self.addCleanup(os.close, stdout_fd_copy) @@ -1539,7 +1536,7 @@ def test_write(self): self.assertEqual(data, message.encode('utf8', 'backslashreplace')) def test_methods(self): - fd = self.get_stdout_fd() + fd = self.STDOUT_FD printer = self.create_printer(fd) self.assertEqual(printer.fileno(), fd) self.assertEqual(printer.isatty(), os.isatty(fd)) @@ -1547,7 +1544,7 @@ def test_methods(self): printer.close() # noop def test_disallow_instantiation(self): - fd = self.get_stdout_fd() + fd = self.STDOUT_FD printer = self.create_printer(fd) support.check_disallow_instantiation(self, type(printer)) From webhook-mailer at python.org Wed Oct 13 08:08:49 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 12:08:49 -0000 Subject: [Python-checkins] bpo-45410: regrtest -W leaves stdout/err FD unchanged (GH-28915) Message-ID: https://github.com/python/cpython/commit/773330773968f211c77abc7b5b525faa7b3c35a2 commit: 773330773968f211c77abc7b5b525faa7b3c35a2 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T14:08:18+02:00 summary: bpo-45410: regrtest -W leaves stdout/err FD unchanged (GH-28915) support.print_warning() now stores the original value of sys.__stderr__ and uses it to log warnings. libregrtest uses the same stream to log unraisable exceptions and uncaught threading exceptions. Partially revert commit dbe213de7ef28712bbfdb9d94a33abb9c33ef0c2: libregrtest no longer replaces sys.__stdout__, sys.__stderr__, and stdout and stderr file descriptors. Remove also a few unused imports in libregrtest. files: M Lib/test/libregrtest/cmdline.py M Lib/test/libregrtest/refleak.py M Lib/test/libregrtest/runtest.py M Lib/test/libregrtest/runtest_mp.py M Lib/test/libregrtest/utils.py M Lib/test/support/__init__.py M Lib/test/test_support.py diff --git a/Lib/test/libregrtest/cmdline.py b/Lib/test/libregrtest/cmdline.py index 29f4ede523b00..6e6ff1201d81c 100644 --- a/Lib/test/libregrtest/cmdline.py +++ b/Lib/test/libregrtest/cmdline.py @@ -1,7 +1,6 @@ import argparse import os import sys -from test import support from test.support import os_helper diff --git a/Lib/test/libregrtest/refleak.py b/Lib/test/libregrtest/refleak.py index b776a2cce647d..1069e2da008ff 100644 --- a/Lib/test/libregrtest/refleak.py +++ b/Lib/test/libregrtest/refleak.py @@ -1,5 +1,4 @@ import os -import re import sys import warnings from inspect import isabstract diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index 31474a222ed09..6fb996a5f947b 100644 --- a/Lib/test/libregrtest/runtest.py +++ b/Lib/test/libregrtest/runtest.py @@ -1,4 +1,3 @@ -import contextlib import faulthandler import functools import gc @@ -6,7 +5,6 @@ import io import os import sys -import tempfile import time import traceback import unittest @@ -175,63 +173,6 @@ def get_abs_module(ns: Namespace, test_name: str) -> str: return 'test.' + test_name - at contextlib.contextmanager -def override_fd(fd, fd2): - fd2_copy = os.dup(fd2) - try: - os.dup2(fd, fd2) - yield - finally: - os.dup2(fd2_copy, fd2) - os.close(fd2_copy) - - -def get_stream_fd(stream): - if stream is None: - return None - try: - return stream.fileno() - except io.UnsupportedOperation: - return None - - - at contextlib.contextmanager -def capture_std_streams(): - """ - Redirect all standard streams to a temporary file: - - * stdout and stderr file descriptors (fd 1 and fd 2) - * sys.stdout, sys.__stdout__ - * sys.stderr, sys.__stderr__ - """ - try: - stderr_fd = sys.stderr.fileno() - except io.UnsupportedOperation: - stderr_fd = None - - # Use a temporary file to support fileno() operation - tmp_file = tempfile.TemporaryFile(mode='w+', - # line buffering - buffering=1, - encoding=sys.stderr.encoding, - errors=sys.stderr.errors) - with contextlib.ExitStack() as stack: - stack.enter_context(tmp_file) - - # Override stdout and stderr file descriptors - tmp_fd = tmp_file.fileno() - for stream in (sys.stdout, sys.stderr): - fd = get_stream_fd(stream) - if fd is not None: - stack.enter_context(override_fd(tmp_fd, fd)) - - # Override sys attributes - for name in ('stdout', 'stderr', '__stdout__', '__stderr__'): - stack.enter_context(support.swap_attr(sys, name, tmp_file)) - - yield tmp_file - - def _runtest(ns: Namespace, test_name: str) -> TestResult: # Handle faulthandler timeout, capture stdout+stderr, XML serialization # and measure time. @@ -252,13 +193,20 @@ def _runtest(ns: Namespace, test_name: str) -> TestResult: if output_on_failure: support.verbose = True + stream = io.StringIO() + orig_stdout = sys.stdout + orig_stderr = sys.stderr output = None - with capture_std_streams() as stream: + try: + sys.stdout = stream + sys.stderr = stream result = _runtest_inner(ns, test_name, display_failure=False) if not isinstance(result, Passed): - stream.seek(0) - output = stream.read() + output = stream.getvalue() + finally: + sys.stdout = orig_stdout + sys.stderr = orig_stderr if output is not None: sys.stderr.write(output) diff --git a/Lib/test/libregrtest/runtest_mp.py b/Lib/test/libregrtest/runtest_mp.py index dea5992feb7b4..f02f56e98bcda 100644 --- a/Lib/test/libregrtest/runtest_mp.py +++ b/Lib/test/libregrtest/runtest_mp.py @@ -1,4 +1,3 @@ -import collections import faulthandler import json import os diff --git a/Lib/test/libregrtest/utils.py b/Lib/test/libregrtest/utils.py index 256f9a4cb6eed..8578a028c78bc 100644 --- a/Lib/test/libregrtest/utils.py +++ b/Lib/test/libregrtest/utils.py @@ -75,7 +75,7 @@ def regrtest_unraisable_hook(unraisable): old_stderr = sys.stderr try: support.flush_std_streams() - sys.stderr = sys.__stderr__ + sys.stderr = support.print_warning.orig_stderr orig_unraisablehook(unraisable) sys.stderr.flush() finally: @@ -98,7 +98,7 @@ def regrtest_threading_excepthook(args): old_stderr = sys.stderr try: support.flush_std_streams() - sys.stderr = sys.__stderr__ + sys.stderr = support.print_warning.orig_stderr orig_threading_excepthook(args) sys.stderr.flush() finally: diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 45801dc317a6a..85fd74126b5f4 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -1177,13 +1177,15 @@ def flush_std_streams(): def print_warning(msg): # bpo-45410: Explicitly flush stdout to keep logs in order flush_std_streams() - # bpo-39983: Print into sys.__stderr__ to display the warning even - # when sys.stderr is captured temporarily by a test - stream = sys.__stderr__ + stream = print_warning.orig_stderr for line in msg.splitlines(): print(f"Warning -- {line}", file=stream) stream.flush() +# bpo-39983: Store the original sys.stderr at Python startup to be able to +# log warnings even if sys.stderr is captured temporarily by a test. +print_warning.orig_stderr = sys.stderr + # Flag used by saved_test_environment of test.libregrtest.save_env, # to check if a test modified the environment. The flag should be set to False diff --git a/Lib/test/test_support.py b/Lib/test/test_support.py index 8b55352b6eecc..d5a1d447f0563 100644 --- a/Lib/test/test_support.py +++ b/Lib/test/test_support.py @@ -469,12 +469,8 @@ def test_reap_children(self): if time.monotonic() > deadline: self.fail("timeout") - old_stderr = sys.__stderr__ - try: - sys.__stderr__ = stderr + with support.swap_attr(support.print_warning, 'orig_stderr', stderr): support.reap_children() - finally: - sys.__stderr__ = old_stderr # Use environment_altered to check if reap_children() found # the child process @@ -674,14 +670,8 @@ def test_fd_count(self): def check_print_warning(self, msg, expected): stderr = io.StringIO() - - old_stderr = sys.__stderr__ - try: - sys.__stderr__ = stderr + with support.swap_attr(support.print_warning, 'orig_stderr', stderr): support.print_warning(msg) - finally: - sys.__stderr__ = old_stderr - self.assertEqual(stderr.getvalue(), expected) def test_print_warning(self): From webhook-mailer at python.org Wed Oct 13 08:09:18 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 12:09:18 -0000 Subject: [Python-checkins] pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921) Message-ID: https://github.com/python/cpython/commit/7cdc2a0f4b785327ad9d55312409a06e554df3d5 commit: 7cdc2a0f4b785327ad9d55312409a06e554df3d5 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T14:09:13+02:00 summary: pycore_pystate.h no longer redefines PyThreadState_GET() (GH-28921) Redefining the PyThreadState_GET() macro in pycore_pystate.h is useless since it doesn't affect files not including it. Either use _PyThreadState_GET() directly, or don't use pycore_pystate.h internal C API. For example, the _testcapi extension don't use the internal C API, but use the public PyThreadState_Get() function instead. Replace PyThreadState_Get() with _PyThreadState_GET(). The _PyThreadState_GET() macro is more efficient than PyThreadState_Get() and PyThreadState_GET() function calls which call fail with a fatal Python error. posixmodule.c and _ctypes extension now include before pycore header files (like pycore_call.h). _PyTraceback_Add() now uses _PyErr_Fetch()/_PyErr_Restore() instead of PyErr_Fetch()/PyErr_Restore(). The _decimal and _xxsubinterpreters extensions are now built with the Py_BUILD_CORE_MODULE macro defined to get access to the internal C API. files: M Include/internal/pycore_call.h M Include/internal/pycore_ceval.h M Include/internal/pycore_pystate.h M Include/pystate.h M Modules/_asynciomodule.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/cfield.c M Modules/_ctypes/stgdict.c M Modules/_decimal/_decimal.c M Modules/_lsprof.c M Modules/_testinternalcapi.c M Modules/_threadmodule.c M Modules/_xxsubinterpretersmodule.c M Modules/posixmodule.c M Objects/typeobject.c M Python/bltinmodule.c M Python/ceval.c M Python/context.c M Python/pylifecycle.c M Python/pystate.c M Python/traceback.c M setup.py diff --git a/Include/internal/pycore_call.h b/Include/internal/pycore_call.h index 28a4194db7484..933d714f13377 100644 --- a/Include/internal/pycore_call.h +++ b/Include/internal/pycore_call.h @@ -8,6 +8,8 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif +#include "pycore_pystate.h" // _PyThreadState_GET() + PyAPI_FUNC(PyObject *) _PyObject_Call_Prepend( PyThreadState *tstate, PyObject *callable, @@ -36,7 +38,7 @@ _PyObject_CallNoArgsTstate(PyThreadState *tstate, PyObject *func) { // Private static inline function variant of public PyObject_CallNoArgs() static inline PyObject * _PyObject_CallNoArgs(PyObject *func) { - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } diff --git a/Include/internal/pycore_ceval.h b/Include/internal/pycore_ceval.h index f2acf10df8c23..53580b99d33ac 100644 --- a/Include/internal/pycore_ceval.h +++ b/Include/internal/pycore_ceval.h @@ -12,7 +12,8 @@ extern "C" { struct pyruntimestate; struct _ceval_runtime_state; -#include "pycore_interp.h" /* PyInterpreterState.eval_frame */ +#include "pycore_interp.h" // PyInterpreterState.eval_frame +#include "pycore_pystate.h" // _PyThreadState_GET() extern void _Py_FinishPendingCalls(PyThreadState *tstate); extern void _PyEval_InitRuntimeState(struct _ceval_runtime_state *); @@ -93,7 +94,7 @@ static inline int _Py_EnterRecursiveCall(PyThreadState *tstate, } static inline int _Py_EnterRecursiveCall_inline(const char *where) { - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); return _Py_EnterRecursiveCall(tstate, where); } @@ -104,7 +105,7 @@ static inline void _Py_LeaveRecursiveCall(PyThreadState *tstate) { } static inline void _Py_LeaveRecursiveCall_inline(void) { - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); _Py_LeaveRecursiveCall(tstate); } diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index aef318989aa6e..1b74e5ffb0473 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -82,7 +82,7 @@ _PyRuntimeState_GetThreadState(_PyRuntimeState *runtime) The caller must hold the GIL. - See also PyThreadState_Get() and PyThreadState_GET(). */ + See also PyThreadState_Get() and _PyThreadState_UncheckedGet(). */ static inline PyThreadState* _PyThreadState_GET(void) { @@ -93,10 +93,6 @@ _PyThreadState_GET(void) #endif } -/* Redefine PyThreadState_GET() as an alias to _PyThreadState_GET() */ -#undef PyThreadState_GET -#define PyThreadState_GET() _PyThreadState_GET() - PyAPI_FUNC(void) _Py_NO_RETURN _Py_FatalError_TstateNULL(const char *func); static inline void diff --git a/Include/pystate.h b/Include/pystate.h index bae440778b261..d81f42a3efe32 100644 --- a/Include/pystate.h +++ b/Include/pystate.h @@ -66,18 +66,10 @@ PyAPI_FUNC(void) PyThreadState_Delete(PyThreadState *); The caller must hold the GIL. - See also PyThreadState_GET() and _PyThreadState_GET(). */ + See also _PyThreadState_UncheckedGet() and _PyThreadState_GET(). */ PyAPI_FUNC(PyThreadState *) PyThreadState_Get(void); -/* Get the current Python thread state. - - Macro using PyThreadState_Get() or _PyThreadState_GET() depending if - pycore_pystate.h is included or not (this header redefines the macro). - - If PyThreadState_Get() is used, issue a fatal error if the current thread - state is NULL. - - See also PyThreadState_Get() and _PyThreadState_GET(). */ +// Alias to PyThreadState_Get() #define PyThreadState_GET() PyThreadState_Get() PyAPI_FUNC(PyThreadState *) PyThreadState_Swap(PyThreadState *); diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index 56079b0277d1a..adc5ff9f796cc 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -1,5 +1,6 @@ #include "Python.h" #include "pycore_pyerrors.h" // _PyErr_ClearExcState() +#include "pycore_pystate.h" // _PyThreadState_GET() #include // offsetof() @@ -225,7 +226,7 @@ get_running_loop(PyObject **loop) { PyObject *rl; - PyThreadState *ts = PyThreadState_Get(); + PyThreadState *ts = _PyThreadState_GET(); uint64_t ts_id = PyThreadState_GetID(ts); if (ts_id == cached_running_holder_tsid && cached_running_holder != NULL) { // Fast path, check the cache. @@ -287,7 +288,7 @@ set_running_loop(PyObject *loop) { PyObject *ts_dict = NULL; - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); if (tstate != NULL) { ts_dict = _PyThreadState_GetDict(tstate); // borrowed } diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index 2567067f0b39f..f8940fdbed242 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -102,12 +102,16 @@ bytes(cdata) #define PY_SSIZE_T_CLEAN #include "Python.h" +// windows.h must be included before pycore internal headers +#ifdef MS_WIN32 +# include +#endif + #include "pycore_call.h" // _PyObject_CallNoArgs() #include "structmember.h" // PyMemberDef #include #ifdef MS_WIN32 -#include #include #ifndef IS_INTRESOURCE #define IS_INTRESOURCE(x) (((size_t)(x) >> 16) == 0) diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index 18b010493628c..c24f04ce3a0f8 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -1,13 +1,15 @@ #include "Python.h" +// windows.h must be included before pycore internal headers +#ifdef MS_WIN32 +# include +#endif + #include "pycore_call.h" // _PyObject_CallNoArgs() #include "frameobject.h" #include #include -#ifdef MS_WIN32 -#include -#endif #include "ctypes.h" /**************************************************************/ diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 6788aee78a2d3..b7585bc8a9fff 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1,11 +1,13 @@ #include "Python.h" +// windows.h must be included before pycore internal headers +#ifdef MS_WIN32 +# include +#endif + #include "pycore_bitutils.h" // _Py_bswap32() #include "pycore_call.h" // _PyObject_CallNoArgs() #include -#ifdef MS_WIN32 -#include -#endif #include "ctypes.h" diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c index ea3c58b88e704..43669d7152a7e 100644 --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -1,9 +1,13 @@ #include "Python.h" +// windows.h must be included before pycore internal headers +#ifdef MS_WIN32 +# include +#endif + #include "pycore_call.h" // _PyObject_CallNoArgs() #include #ifdef MS_WIN32 -#include -#include +# include #endif #include "ctypes.h" diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index e2979a577c96c..dd876f200365d 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -27,6 +27,7 @@ #include +#include "pycore_pystate.h" // _PyThreadState_GET() #include "longintrepr.h" #include "complexobject.h" #include "mpdecimal.h" @@ -1511,18 +1512,20 @@ static PyGetSetDef context_getsets [] = static PyObject * current_context_from_dict(void) { - PyObject *dict; - PyObject *tl_context; - PyThreadState *tstate; + PyThreadState *tstate = _PyThreadState_GET(); +#ifdef Py_DEBUG + // The caller must hold the GIL + _Py_EnsureTstateNotNULL(tstate); +#endif - dict = PyThreadState_GetDict(); + PyObject *dict = _PyThreadState_GetDict(tstate); if (dict == NULL) { PyErr_SetString(PyExc_RuntimeError, "cannot get thread state"); return NULL; } - tl_context = PyDict_GetItemWithError(dict, tls_context_key); + PyObject *tl_context = PyDict_GetItemWithError(dict, tls_context_key); if (tl_context != NULL) { /* We already have a thread local context. */ CONTEXT_CHECK(tl_context); @@ -1548,11 +1551,8 @@ current_context_from_dict(void) /* Cache the context of the current thread, assuming that it * will be accessed several times before a thread switch. */ - tstate = PyThreadState_GET(); - if (tstate) { - cached_context = (PyDecContextObject *)tl_context; - cached_context->tstate = tstate; - } + cached_context = (PyDecContextObject *)tl_context; + cached_context->tstate = tstate; /* Borrowed reference with refcount==1 */ return tl_context; @@ -1562,9 +1562,7 @@ current_context_from_dict(void) static PyObject * current_context(void) { - PyThreadState *tstate; - - tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); if (cached_context && cached_context->tstate == tstate) { return (PyObject *)cached_context; } diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 097d0eff2602d..2e27afcea1b79 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -1,5 +1,6 @@ #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_pystate.h" // _PyThreadState_GET() #include "rotatingtree.h" /************************************************************/ @@ -672,7 +673,7 @@ profiler_enable(ProfilerObject *self, PyObject *args, PyObject *kwds) return NULL; } - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); if (_PyEval_SetProfile(tstate, profiler_callback, (PyObject*)self) < 0) { return NULL; } @@ -706,7 +707,7 @@ Stop collecting profiling information.\n\ static PyObject* profiler_disable(ProfilerObject *self, PyObject* noarg) { - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); if (_PyEval_SetProfile(tstate, NULL, NULL) < 0) { return NULL; } @@ -743,7 +744,7 @@ static void profiler_dealloc(ProfilerObject *op) { if (op->flags & POF_ENABLED) { - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); if (_PyEval_SetProfile(tstate, NULL, NULL) < 0) { PyErr_WriteUnraisable((PyObject *)op); } diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index d5616fd59c6e5..3ba939651a417 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -18,7 +18,8 @@ #include "pycore_hashtable.h" // _Py_hashtable_new() #include "pycore_initconfig.h" // _Py_GetConfigsAsDict() #include "pycore_interp.h" // _PyInterpreterState_GetConfigCopy() -#include "pycore_pyerrors.h" // _Py_UTF8_Edit_Cost() +#include "pycore_pyerrors.h" // _Py_UTF8_Edit_Cost() +#include "pycore_pystate.h" // _PyThreadState_GET() static PyObject * @@ -31,7 +32,7 @@ get_configs(PyObject *self, PyObject *Py_UNUSED(args)) static PyObject* get_recursion_depth(PyObject *self, PyObject *Py_UNUSED(args)) { - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); /* subtract one to ignore the frame of the get_recursion_depth() call */ return PyLong_FromLong(tstate->recursion_depth - 1); diff --git a/Modules/_threadmodule.c b/Modules/_threadmodule.c index ff858386b426f..39b116afcaa3e 100644 --- a/Modules/_threadmodule.c +++ b/Modules/_threadmodule.c @@ -1319,7 +1319,7 @@ static PyObject * thread__set_sentinel(PyObject *module, PyObject *Py_UNUSED(ignored)) { PyObject *wr; - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); lockobject *lock; if (tstate->on_delete_data != NULL) { diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index 11b55f7f0c7d1..939114e60cae9 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -5,6 +5,7 @@ #include "Python.h" #include "frameobject.h" #include "pycore_frame.h" +#include "pycore_pystate.h" // _PyThreadState_GET() #include "interpreteridobject.h" @@ -2017,7 +2018,7 @@ interp_create(PyObject *self, PyObject *args, PyObject *kwds) } // Create and initialize the new interpreter. - PyThreadState *save_tstate = PyThreadState_Get(); + PyThreadState *save_tstate = _PyThreadState_GET(); // XXX Possible GILState issues? PyThreadState *tstate = _Py_NewInterpreter(isolated); PyThreadState_Swap(save_tstate); diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index ada1a5865aae7..9c174ee6b18df 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -10,31 +10,25 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pycore_call.h" // _PyObject_CallNoArgs() -#include "pycore_fileutils.h" // _Py_closerange() -#include "pycore_moduleobject.h" // _PyModule_GetState() +// Include before pycore internal headers. FSCTL_GET_REPARSE_POINT +// is not exported by if the WIN32_LEAN_AND_MEAN macro is defined, +// whereas pycore_condvar.h defines the WIN32_LEAN_AND_MEAN macro. #ifdef MS_WINDOWS - /* include early to avoid conflict with pycore_condvar.h: - - #define WIN32_LEAN_AND_MEAN - #include - - FSCTL_GET_REPARSE_POINT is not exported with WIN32_LEAN_AND_MEAN. */ # include # include #endif -#if !defined(EX_OK) && defined(EXIT_SUCCESS) -#define EX_OK EXIT_SUCCESS -#endif - #ifdef __VXWORKS__ # include "pycore_bitutils.h" // _Py_popcount32() #endif +#include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_fileutils.h" // _Py_closerange() +#include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_ceval.h" // _PyEval_ReInitThreads() #include "pycore_import.h" // _PyImport_ReInitLock() #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() #include "pycore_pystate.h" // _PyInterpreterState_GET() + #include "structmember.h" // PyMemberDef #ifndef MS_WINDOWS # include "posixmodule.h" @@ -42,6 +36,10 @@ # include "winreparse.h" #endif +#if !defined(EX_OK) && defined(EXIT_SUCCESS) +# define EX_OK EXIT_SUCCESS +#endif + /* On android API level 21, 'AT_EACCESS' is not declared although * HAVE_FACCESSAT is defined. */ #ifdef __ANDROID__ diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 0f56e59bc5185..544e8a4aa2211 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -3332,7 +3332,7 @@ type_vectorcall(PyObject *metatype, PyObject *const *args, } /* In other (much less common) cases, fall back to more flexible calling conventions. */ - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); return _PyObject_MakeTpCall(tstate, metatype, args, nargs, kwnames); } diff --git a/Python/bltinmodule.c b/Python/bltinmodule.c index d07ba38b698c6..9e3b25c59a550 100644 --- a/Python/bltinmodule.c +++ b/Python/bltinmodule.c @@ -213,7 +213,7 @@ builtin___build_class__(PyObject *self, PyObject *const *args, Py_ssize_t nargs, goto error; } PyFrameConstructor *f = PyFunction_AS_FRAME_CONSTRUCTOR(func); - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); cell = _PyEval_Vector(tstate, f, ns, NULL, 0, NULL); if (cell != NULL) { if (bases != orig_bases) { diff --git a/Python/ceval.c b/Python/ceval.c index 2d617a6364880..f4cacd84b66cb 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1064,7 +1064,7 @@ static int unpack_iterable(PyThreadState *, PyObject *, int, int, PyObject **); PyObject * PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals) { - PyThreadState *tstate = PyThreadState_GET(); + PyThreadState *tstate = _PyThreadState_GET(); if (locals == NULL) { locals = globals; } diff --git a/Python/context.c b/Python/context.c index 9560fb3f6676e..ad47992d9e3cd 100644 --- a/Python/context.c +++ b/Python/context.c @@ -728,7 +728,7 @@ static int contextvar_set(PyContextVar *var, PyObject *val) { var->var_cached = NULL; - PyThreadState *ts = PyThreadState_Get(); + PyThreadState *ts = _PyThreadState_GET(); PyContext *ctx = context_get(); if (ctx == NULL) { diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index bfddc1922d25a..c67a9b7fbfa2d 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -472,7 +472,7 @@ interpreter_update_config(PyThreadState *tstate, int only_update_path_config) int _PyInterpreterState_SetConfig(const PyConfig *src_config) { - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); int res = -1; PyConfig config; diff --git a/Python/pystate.c b/Python/pystate.c index f02de926c6f02..ee9507c7dd80b 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1671,8 +1671,11 @@ _check_xidata(PyThreadState *tstate, _PyCrossInterpreterData *data) int _PyObject_GetCrossInterpreterData(PyObject *obj, _PyCrossInterpreterData *data) { - // PyThreadState_Get() aborts if tstate is NULL. - PyThreadState *tstate = PyThreadState_Get(); + PyThreadState *tstate = _PyThreadState_GET(); +#ifdef Py_DEBUG + // The caller must hold the GIL + _Py_EnsureTstateNotNULL(tstate); +#endif PyInterpreterState *interp = tstate->interp; // Reset data before re-populating. diff --git a/Python/traceback.c b/Python/traceback.c index 06b40bbbdc9f8..3ea1db578b43a 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -6,11 +6,13 @@ #include "code.h" // PyCode_Addr2Line etc #include "pycore_interp.h" // PyInterpreterState.gc #include "frameobject.h" // PyFrame_GetBack() -#include "pycore_frame.h" // _PyFrame_GetCode() -#include "pycore_pyarena.h" // _PyArena_Free() #include "pycore_ast.h" // asdl_seq_* #include "pycore_compile.h" // _PyAST_Optimize +#include "pycore_frame.h" // _PyFrame_GetCode() #include "pycore_parser.h" // _PyParser_ASTFromString +#include "pycore_pyarena.h" // _PyArena_Free() +#include "pycore_pyerrors.h" // _PyErr_Fetch() +#include "pycore_pystate.h" // _PyThreadState_GET() #include "../Parser/pegen.h" // _PyPegen_byte_offset_to_character_offset() #include "structmember.h" // PyMemberDef #include "osdefs.h" // SEP @@ -267,11 +269,12 @@ void _PyTraceback_Add(const char *funcname, const char *filename, int lineno) PyCodeObject *code; PyFrameObject *frame; PyObject *exc, *val, *tb; + PyThreadState *tstate = _PyThreadState_GET(); /* Save and clear the current exception. Python functions must not be called with an exception set. Calling Python functions happens when the codec of the filesystem encoding is implemented in pure Python. */ - PyErr_Fetch(&exc, &val, &tb); + _PyErr_Fetch(tstate, &exc, &val, &tb); globals = PyDict_New(); if (!globals) @@ -281,14 +284,14 @@ void _PyTraceback_Add(const char *funcname, const char *filename, int lineno) Py_DECREF(globals); goto error; } - frame = PyFrame_New(PyThreadState_Get(), code, globals, NULL); + frame = PyFrame_New(tstate, code, globals, NULL); Py_DECREF(globals); Py_DECREF(code); if (!frame) goto error; frame->f_lineno = lineno; - PyErr_Restore(exc, val, tb); + _PyErr_Restore(tstate, exc, val, tb); PyTraceBack_Here(frame); Py_DECREF(frame); return; diff --git a/setup.py b/setup.py index c6290eebed30f..039c96b0e4d7b 100644 --- a/setup.py +++ b/setup.py @@ -1000,7 +1000,8 @@ def detect_simple_extensions(self): self.add(Extension('syslog', ['syslogmodule.c'])) # Python interface to subinterpreter C-API. - self.add(Extension('_xxsubinterpreters', ['_xxsubinterpretersmodule.c'])) + self.add(Extension('_xxsubinterpreters', ['_xxsubinterpretersmodule.c'], + extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) # # Here ends the simple stuff. From here on, modules need certain @@ -2310,7 +2311,7 @@ def detect_ctypes(self): def detect_decimal(self): # Stefan Krah's _decimal module - extra_compile_args = [] + extra_compile_args = ['-DPy_BUILD_CORE_MODULE'] undef_macros = [] if '--with-system-libmpdec' in sysconfig.get_config_var("CONFIG_ARGS"): include_dirs = [] From webhook-mailer at python.org Wed Oct 13 09:03:44 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 13:03:44 -0000 Subject: [Python-checkins] bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) Message-ID: https://github.com/python/cpython/commit/97308dfcdc0696e0b116c37386e2ff4d72e6c3f4 commit: 97308dfcdc0696e0b116c37386e2ff4d72e6c3f4 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T15:03:35+02:00 summary: bpo-45434: Move _Py_BEGIN_SUPPRESS_IPH to pycore_fileutils.h (GH-28922) files: M Include/internal/pycore_fileutils.h M Include/pyport.h M Modules/_io/fileio.c M Modules/_io/winconsoleio.c M Modules/signalmodule.c M Modules/timemodule.c M PC/getpathp.c M PC/msvcrtmodule.c M Parser/myreadline.c M Python/traceback.c diff --git a/Include/internal/pycore_fileutils.h b/Include/internal/pycore_fileutils.h index 8491ed9b5ffe2..2316a978294e2 100644 --- a/Include/internal/pycore_fileutils.h +++ b/Include/internal/pycore_fileutils.h @@ -80,6 +80,25 @@ extern int _Py_add_relfile(wchar_t *dirname, const wchar_t *relfile, size_t bufsize); +// Macros to protect CRT calls against instant termination when passed an +// invalid parameter (bpo-23524). IPH stands for Invalid Parameter Handler. +// Usage: +// +// _Py_BEGIN_SUPPRESS_IPH +// ... +// _Py_END_SUPPRESS_IPH +#if defined _MSC_VER && _MSC_VER >= 1900 + extern _invalid_parameter_handler _Py_silent_invalid_parameter_handler; +# define _Py_BEGIN_SUPPRESS_IPH \ + { _invalid_parameter_handler _Py_old_handler = \ + _set_thread_local_invalid_parameter_handler(_Py_silent_invalid_parameter_handler); +# define _Py_END_SUPPRESS_IPH \ + _set_thread_local_invalid_parameter_handler(_Py_old_handler); } +#else +# define _Py_BEGIN_SUPPRESS_IPH +# define _Py_END_SUPPRESS_IPH +#endif /* _MSC_VER >= 1900 */ + #ifdef __cplusplus } #endif diff --git a/Include/pyport.h b/Include/pyport.h index a8a2d6d0d9d92..6e4e98020cb1f 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -712,26 +712,6 @@ extern char * _getpty(int *, int, mode_t, int); # define PY_LITTLE_ENDIAN 1 #endif -#ifdef Py_BUILD_CORE -/* - * Macros to protect CRT calls against instant termination when passed an - * invalid parameter (issue23524). - */ -#if defined _MSC_VER && _MSC_VER >= 1900 - -extern _invalid_parameter_handler _Py_silent_invalid_parameter_handler; -#define _Py_BEGIN_SUPPRESS_IPH { _invalid_parameter_handler _Py_old_handler = \ - _set_thread_local_invalid_parameter_handler(_Py_silent_invalid_parameter_handler); -#define _Py_END_SUPPRESS_IPH _set_thread_local_invalid_parameter_handler(_Py_old_handler); } - -#else - -#define _Py_BEGIN_SUPPRESS_IPH -#define _Py_END_SUPPRESS_IPH - -#endif /* _MSC_VER >= 1900 */ -#endif /* Py_BUILD_CORE */ - #ifdef __ANDROID__ /* The Android langinfo.h header is not used. */ # undef HAVE_LANGINFO_H diff --git a/Modules/_io/fileio.c b/Modules/_io/fileio.c index b9856b3b63165..dd215e8939944 100644 --- a/Modules/_io/fileio.c +++ b/Modules/_io/fileio.c @@ -2,7 +2,8 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pycore_object.h" +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH +#include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "structmember.h" // PyMemberDef #include #ifdef HAVE_SYS_TYPES_H diff --git a/Modules/_io/winconsoleio.c b/Modules/_io/winconsoleio.c index 460f2d3fa071a..4002d28fc9b35 100644 --- a/Modules/_io/winconsoleio.c +++ b/Modules/_io/winconsoleio.c @@ -8,7 +8,8 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pycore_object.h" +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH +#include "pycore_object.h" // _PyObject_GC_UNTRACK() #ifdef MS_WINDOWS diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 6568a4d5aa70e..09f4aed9d5ca9 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -7,6 +7,7 @@ #include "pycore_atomic.h" // _Py_atomic_int #include "pycore_call.h" // _PyObject_Call() #include "pycore_ceval.h" // _PyEval_SignalReceived() +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_frame.h" // InterpreterFrame #include "pycore_moduleobject.h" // _PyModule_GetState() #include "pycore_pyerrors.h" // _PyErr_SetString() diff --git a/Modules/timemodule.c b/Modules/timemodule.c index 4639afa590a4f..ca8d62d5cc938 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -1,33 +1,28 @@ /* Time module */ #include "Python.h" +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include #ifdef HAVE_SYS_TIMES_H -#include +# include #endif - #ifdef HAVE_SYS_TYPES_H -#include +# include #endif - #if defined(HAVE_SYS_RESOURCE_H) -#include +# include #endif - #ifdef QUICKWIN -#include +# include #endif - #if defined(HAVE_PTHREAD_H) # include #endif - #if defined(_AIX) # include #endif - #if defined(__WATCOMC__) && !defined(__QNX__) # include #else @@ -38,17 +33,17 @@ #endif /* !__WATCOMC__ || __QNX__ */ #ifdef _Py_MEMORY_SANITIZER -# include +# include #endif #ifdef _MSC_VER -#define _Py_timezone _timezone -#define _Py_daylight _daylight -#define _Py_tzname _tzname +# define _Py_timezone _timezone +# define _Py_daylight _daylight +# define _Py_tzname _tzname #else -#define _Py_timezone timezone -#define _Py_daylight daylight -#define _Py_tzname tzname +# define _Py_timezone timezone +# define _Py_daylight daylight +# define _Py_tzname tzname #endif #if defined(__APPLE__ ) && defined(__has_builtin) @@ -60,8 +55,10 @@ # define HAVE_CLOCK_GETTIME_RUNTIME 1 #endif + #define SEC_TO_NS (1000 * 1000 * 1000) + /* Forward declarations */ static int pysleep(_PyTime_t timeout); diff --git a/PC/getpathp.c b/PC/getpathp.c index 79569bb554457..549353d78b116 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -80,9 +80,9 @@ #include "Python.h" +#include "pycore_fileutils.h" // _Py_add_relfile() #include "pycore_initconfig.h" // PyStatus #include "pycore_pathconfig.h" // _PyPathConfig -#include "pycore_fileutils.h" // _Py_add_relfile() #include "osdefs.h" // SEP, ALTSEP #include diff --git a/PC/msvcrtmodule.c b/PC/msvcrtmodule.c index 0591497871cad..1f78d99c790ff 100644 --- a/PC/msvcrtmodule.c +++ b/PC/msvcrtmodule.c @@ -17,6 +17,7 @@ ***********************************************************/ #include "Python.h" +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "malloc.h" #include #include diff --git a/Parser/myreadline.c b/Parser/myreadline.c index e5e2fb1099d03..b10d306255bb6 100644 --- a/Parser/myreadline.c +++ b/Parser/myreadline.c @@ -10,6 +10,7 @@ */ #include "Python.h" +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_pystate.h" // _PyThreadState_GET() #ifdef MS_WINDOWS # define WIN32_LEAN_AND_MEAN diff --git a/Python/traceback.c b/Python/traceback.c index 3ea1db578b43a..ffa7c3494eafc 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -4,11 +4,12 @@ #include "Python.h" #include "code.h" // PyCode_Addr2Line etc -#include "pycore_interp.h" // PyInterpreterState.gc #include "frameobject.h" // PyFrame_GetBack() #include "pycore_ast.h" // asdl_seq_* #include "pycore_compile.h" // _PyAST_Optimize +#include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH #include "pycore_frame.h" // _PyFrame_GetCode() +#include "pycore_interp.h" // PyInterpreterState.gc #include "pycore_parser.h" // _PyParser_ASTFromString #include "pycore_pyarena.h" // _PyArena_Free() #include "pycore_pyerrors.h" // _PyErr_Fetch() @@ -17,7 +18,7 @@ #include "structmember.h" // PyMemberDef #include "osdefs.h" // SEP #ifdef HAVE_FCNTL_H -#include +# include #endif #define OFF(x) offsetof(PyTracebackObject, x) From webhook-mailer at python.org Wed Oct 13 09:19:43 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 13 Oct 2021 13:19:43 -0000 Subject: [Python-checkins] bpo-45340: Don't create object dictionaries unless actually needed (GH-28802) Message-ID: https://github.com/python/cpython/commit/a8b9350964f43cb648c98c179c8037fbf3ff8a7d commit: a8b9350964f43cb648c98c179c8037fbf3ff8a7d branch: main author: Mark Shannon committer: markshannon date: 2021-10-13T14:19:34+01:00 summary: bpo-45340: Don't create object dictionaries unless actually needed (GH-28802) * Never change types' cached keys. It could invalidate inline attribute objects. * Lazily create object dictionaries. * Update specialization of LOAD/STORE_ATTR. * Don't update shared keys version for deletion of value. * Update gdb support to handle instance values. * Rename SPLIT_KEYS opcodes to INSTANCE_VALUE. files: A Misc/NEWS.d/next/Core and Builtins/2021-10-08-09-47-38.bpo-45340.ukHgDb.rst M Include/cpython/object.h M Include/internal/pycore_dict.h M Include/internal/pycore_object.h M Include/object.h M Include/opcode.h M Lib/opcode.py M Lib/test/test_descr.py M Lib/test/test_dict.py M Lib/test/test_gc.py M Lib/test/test_sys.py M Objects/dictobject.c M Objects/object.c M Objects/typeobject.c M Python/ceval.c M Python/opcode_targets.h M Python/specialize.c M Tools/gdb/libpython.py diff --git a/Include/cpython/object.h b/Include/cpython/object.h index 5ae6f367c6048..849d5aa6a2c72 100644 --- a/Include/cpython/object.h +++ b/Include/cpython/object.h @@ -270,6 +270,7 @@ struct _typeobject { destructor tp_finalize; vectorcallfunc tp_vectorcall; + Py_ssize_t tp_inline_values_offset; }; /* The *real* layout of a type object when allocated on the heap */ diff --git a/Include/internal/pycore_dict.h b/Include/internal/pycore_dict.h index d37ef71bbcad3..13cb7ccc7b819 100644 --- a/Include/internal/pycore_dict.h +++ b/Include/internal/pycore_dict.h @@ -101,6 +101,8 @@ extern uint64_t _pydict_global_version; #define DICT_NEXT_VERSION() (++_pydict_global_version) +PyObject *_PyObject_MakeDictFromInstanceAttributes(PyObject *obj, PyDictValues *values); + #ifdef __cplusplus } #endif diff --git a/Include/internal/pycore_object.h b/Include/internal/pycore_object.h index 82dddf119a47a..3c126aaef1187 100644 --- a/Include/internal/pycore_object.h +++ b/Include/internal/pycore_object.h @@ -181,6 +181,16 @@ extern int _Py_CheckSlotResult( extern PyObject* _PyType_AllocNoTrack(PyTypeObject *type, Py_ssize_t nitems); extern int _PyObject_InitializeDict(PyObject *obj); +extern int _PyObject_StoreInstanceAttribute(PyObject *obj, PyDictValues *values, + PyObject *name, PyObject *value); +PyObject * _PyObject_GetInstanceAttribute(PyObject *obj, PyDictValues *values, + PyObject *name); +PyDictValues ** _PyObject_ValuesPointer(PyObject *); +PyObject ** _PyObject_DictPointer(PyObject *); +int _PyObject_VisitInstanceAttributes(PyObject *self, visitproc visit, void *arg); +void _PyObject_ClearInstanceAttributes(PyObject *self); +void _PyObject_FreeInstanceAttributes(PyObject *self); +int _PyObject_IsInstanceDictEmpty(PyObject *); #ifdef __cplusplus } diff --git a/Include/object.h b/Include/object.h index c3062cb01c8e0..7f050b80aaff1 100644 --- a/Include/object.h +++ b/Include/object.h @@ -333,6 +333,7 @@ given type object has a specified feature. */ #ifndef Py_LIMITED_API + /* Set if instances of the type object are treated as sequences for pattern matching */ #define Py_TPFLAGS_SEQUENCE (1 << 5) /* Set if instances of the type object are treated as mappings for pattern matching */ diff --git a/Include/opcode.h b/Include/opcode.h index 8817a4d650fd1..15f722630dc5d 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -147,7 +147,7 @@ extern "C" { #define BINARY_SUBSCR_DICT 39 #define JUMP_ABSOLUTE_QUICK 40 #define LOAD_ATTR_ADAPTIVE 41 -#define LOAD_ATTR_SPLIT_KEYS 42 +#define LOAD_ATTR_INSTANCE_VALUE 42 #define LOAD_ATTR_WITH_HINT 43 #define LOAD_ATTR_SLOT 44 #define LOAD_ATTR_MODULE 45 @@ -158,15 +158,16 @@ extern "C" { #define LOAD_METHOD_CACHED 80 #define LOAD_METHOD_CLASS 81 #define LOAD_METHOD_MODULE 87 -#define STORE_ATTR_ADAPTIVE 88 -#define STORE_ATTR_SPLIT_KEYS 120 -#define STORE_ATTR_SLOT 122 -#define STORE_ATTR_WITH_HINT 123 -#define LOAD_FAST__LOAD_FAST 127 -#define STORE_FAST__LOAD_FAST 128 -#define LOAD_FAST__LOAD_CONST 134 -#define LOAD_CONST__LOAD_FAST 140 -#define STORE_FAST__STORE_FAST 143 +#define LOAD_METHOD_NO_DICT 88 +#define STORE_ATTR_ADAPTIVE 120 +#define STORE_ATTR_INSTANCE_VALUE 122 +#define STORE_ATTR_SLOT 123 +#define STORE_ATTR_WITH_HINT 127 +#define LOAD_FAST__LOAD_FAST 128 +#define STORE_FAST__LOAD_FAST 134 +#define LOAD_FAST__LOAD_CONST 140 +#define LOAD_CONST__LOAD_FAST 143 +#define STORE_FAST__STORE_FAST 149 #define DO_TRACING 255 #ifdef NEED_OPCODE_JUMP_TABLES static uint32_t _PyOpcode_RelativeJump[8] = { diff --git a/Lib/opcode.py b/Lib/opcode.py index 5d35674688875..efd6aefccc571 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -231,7 +231,7 @@ def jabs_op(name, op): "BINARY_SUBSCR_DICT", "JUMP_ABSOLUTE_QUICK", "LOAD_ATTR_ADAPTIVE", - "LOAD_ATTR_SPLIT_KEYS", + "LOAD_ATTR_INSTANCE_VALUE", "LOAD_ATTR_WITH_HINT", "LOAD_ATTR_SLOT", "LOAD_ATTR_MODULE", @@ -242,8 +242,9 @@ def jabs_op(name, op): "LOAD_METHOD_CACHED", "LOAD_METHOD_CLASS", "LOAD_METHOD_MODULE", + "LOAD_METHOD_NO_DICT", "STORE_ATTR_ADAPTIVE", - "STORE_ATTR_SPLIT_KEYS", + "STORE_ATTR_INSTANCE_VALUE", "STORE_ATTR_SLOT", "STORE_ATTR_WITH_HINT", # Super instructions diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index af7848c0b1b3c..a5404b30d2459 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -5500,17 +5500,19 @@ class A: class B(A): pass + #Shrink keys by repeatedly creating instances + [(A(), B()) for _ in range(20)] + a, b = A(), B() self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({"a":1})) - # Initial hash table can contain at most 5 elements. + # Initial hash table can contain only one or two elements. # Set 6 attributes to cause internal resizing. a.x, a.y, a.z, a.w, a.v, a.u = range(6) self.assertNotEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(b))) a2 = A() - self.assertEqual(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) - self.assertLess(sys.getsizeof(vars(a)), sys.getsizeof({"a":1})) - b.u, b.v, b.w, b.t, b.s, b.r = range(6) + self.assertGreater(sys.getsizeof(vars(a)), sys.getsizeof(vars(a2))) + self.assertLess(sys.getsizeof(vars(a2)), sys.getsizeof({"a":1})) self.assertLess(sys.getsizeof(vars(b)), sys.getsizeof({"a":1})) diff --git a/Lib/test/test_dict.py b/Lib/test/test_dict.py index 40143757fd36a..b43c83abd0eeb 100644 --- a/Lib/test/test_dict.py +++ b/Lib/test/test_dict.py @@ -994,8 +994,8 @@ class C: @support.cpython_only def test_splittable_setdefault(self): - """split table must be combined when setdefault() - breaks insertion order""" + """split table must keep correct insertion + order when attributes are adding using setdefault()""" a, b = self.make_shared_key_dict(2) a['a'] = 1 @@ -1005,7 +1005,6 @@ def test_splittable_setdefault(self): size_b = sys.getsizeof(b) b['a'] = 1 - self.assertGreater(size_b, size_a) self.assertEqual(list(a), ['x', 'y', 'z', 'a', 'b']) self.assertEqual(list(b), ['x', 'y', 'z', 'b', 'a']) diff --git a/Lib/test/test_gc.py b/Lib/test/test_gc.py index 6c28b2b677cee..52948f1c7bde5 100644 --- a/Lib/test/test_gc.py +++ b/Lib/test/test_gc.py @@ -444,7 +444,7 @@ def __getattr__(self, someattribute): # 0, thus mutating the trash graph as a side effect of merely asking # whether __del__ exists. This used to (before 2.3b1) crash Python. # Now __getattr__ isn't called. - self.assertEqual(gc.collect(), 4) + self.assertEqual(gc.collect(), 2) self.assertEqual(len(gc.garbage), garbagelen) def test_boom2(self): @@ -471,7 +471,7 @@ def __getattr__(self, someattribute): # there isn't a second time, so this simply cleans up the trash cycle. # We expect a, b, a.__dict__ and b.__dict__ (4 objects) to get # reclaimed this way. - self.assertEqual(gc.collect(), 4) + self.assertEqual(gc.collect(), 2) self.assertEqual(len(gc.garbage), garbagelen) def test_boom_new(self): @@ -491,7 +491,7 @@ def __getattr__(self, someattribute): gc.collect() garbagelen = len(gc.garbage) del a, b - self.assertEqual(gc.collect(), 4) + self.assertEqual(gc.collect(), 2) self.assertEqual(len(gc.garbage), garbagelen) def test_boom2_new(self): @@ -513,7 +513,7 @@ def __getattr__(self, someattribute): gc.collect() garbagelen = len(gc.garbage) del a, b - self.assertEqual(gc.collect(), 4) + self.assertEqual(gc.collect(), 2) self.assertEqual(len(gc.garbage), garbagelen) def test_get_referents(self): @@ -943,8 +943,8 @@ def getstats(): A() t = gc.collect() c, nc = getstats() - self.assertEqual(t, 2*N) # instance object & its dict - self.assertEqual(c - oldc, 2*N) + self.assertEqual(t, N) # instance objects + self.assertEqual(c - oldc, N) self.assertEqual(nc - oldnc, 0) # But Z() is not actually collected. @@ -964,8 +964,8 @@ def getstats(): Z() t = gc.collect() c, nc = getstats() - self.assertEqual(t, 2*N) - self.assertEqual(c - oldc, 2*N) + self.assertEqual(t, N) + self.assertEqual(c - oldc, N) self.assertEqual(nc - oldnc, 0) # The A() trash should have been reclaimed already but the @@ -974,8 +974,8 @@ def getstats(): zs.clear() t = gc.collect() c, nc = getstats() - self.assertEqual(t, 4) - self.assertEqual(c - oldc, 4) + self.assertEqual(t, 2) + self.assertEqual(c - oldc, 2) self.assertEqual(nc - oldnc, 0) gc.enable() @@ -1128,8 +1128,7 @@ def test_collect_generation(self): @cpython_only def test_collect_garbage(self): self.preclean() - # Each of these cause four objects to be garbage: Two - # Uncollectables and their instance dicts. + # Each of these cause two objects to be garbage: Uncollectable() Uncollectable() C1055820(666) @@ -1138,8 +1137,8 @@ def test_collect_garbage(self): if v[1] != "stop": continue info = v[2] - self.assertEqual(info["collected"], 2) - self.assertEqual(info["uncollectable"], 8) + self.assertEqual(info["collected"], 1) + self.assertEqual(info["uncollectable"], 4) # We should now have the Uncollectables in gc.garbage self.assertEqual(len(gc.garbage), 4) @@ -1156,7 +1155,7 @@ def test_collect_garbage(self): continue info = v[2] self.assertEqual(info["collected"], 0) - self.assertEqual(info["uncollectable"], 4) + self.assertEqual(info["uncollectable"], 2) # Uncollectables should be gone self.assertEqual(len(gc.garbage), 0) diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 93e39bc1836e0..2ce40fcde9eb2 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1409,7 +1409,7 @@ def delx(self): del self.__x check((1,2,3), vsize('') + 3*self.P) # type # static type: PyTypeObject - fmt = 'P2nPI13Pl4Pn9Pn11PIPP' + fmt = 'P2nPI13Pl4Pn9Pn12PIPP' s = vsize(fmt) check(int, s) # class @@ -1422,15 +1422,15 @@ def delx(self): del self.__x '5P') class newstyleclass(object): pass # Separate block for PyDictKeysObject with 8 keys and 5 entries - check(newstyleclass, s + calcsize(DICT_KEY_STRUCT_FORMAT) + 8 + 5*calcsize("n2P")) + check(newstyleclass, s + calcsize(DICT_KEY_STRUCT_FORMAT) + 32 + 21*calcsize("n2P")) # dict with shared keys - check(newstyleclass().__dict__, size('nQ2P') + 5*self.P) + check(newstyleclass().__dict__, size('nQ2P') + 15*self.P) o = newstyleclass() o.a = o.b = o.c = o.d = o.e = o.f = o.g = o.h = 1 # Separate block for PyDictKeysObject with 16 keys and 10 entries - check(newstyleclass, s + calcsize(DICT_KEY_STRUCT_FORMAT) + 16 + 10*calcsize("n2P")) + check(newstyleclass, s + calcsize(DICT_KEY_STRUCT_FORMAT) + 32 + 21*calcsize("n2P")) # dict with shared keys - check(newstyleclass().__dict__, size('nQ2P') + 10*self.P) + check(newstyleclass().__dict__, size('nQ2P') + 13*self.P) # unicode # each tuple contains a string and its expected character size # don't put any static strings here, as they may contain diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-08-09-47-38.bpo-45340.ukHgDb.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-08-09-47-38.bpo-45340.ukHgDb.rst new file mode 100644 index 0000000000000..7760773d4fb20 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-08-09-47-38.bpo-45340.ukHgDb.rst @@ -0,0 +1,3 @@ +Object attributes are held in an array instead of a dictionary. An object's +dictionary are created lazily, only when needed. Reduces the memory +consumption of a typical Python object by about 30%. Patch by Mark Shannon. diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 60470bf66b161..3d6e4c1e17e1f 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -634,7 +634,7 @@ new_values(Py_ssize_t size) /* Consumes a reference to the keys object */ static PyObject * -new_dict(PyDictKeysObject *keys, PyDictValues *values) +new_dict(PyDictKeysObject *keys, PyDictValues *values, Py_ssize_t used, int free_values_on_failure) { PyDictObject *mp; assert(keys != NULL); @@ -653,7 +653,7 @@ new_dict(PyDictKeysObject *keys, PyDictValues *values) mp = PyObject_GC_New(PyDictObject, &PyDict_Type); if (mp == NULL) { dictkeys_decref(keys); - if (values != empty_values) { + if (free_values_on_failure) { free_values(values); } return NULL; @@ -661,12 +661,18 @@ new_dict(PyDictKeysObject *keys, PyDictValues *values) } mp->ma_keys = keys; mp->ma_values = values; - mp->ma_used = 0; + mp->ma_used = used; mp->ma_version_tag = DICT_NEXT_VERSION(); ASSERT_CONSISTENT(mp); return (PyObject *)mp; } +static inline Py_ssize_t +shared_keys_usable_size(PyDictKeysObject *keys) +{ + return keys->dk_nentries + keys->dk_usable; +} + /* Consumes a reference to the keys object */ static PyObject * new_dict_with_shared_keys(PyDictKeysObject *keys) @@ -674,7 +680,7 @@ new_dict_with_shared_keys(PyDictKeysObject *keys) PyDictValues *values; Py_ssize_t i, size; - size = USABLE_FRACTION(DK_SIZE(keys)); + size = shared_keys_usable_size(keys); values = new_values(size); if (values == NULL) { dictkeys_decref(keys); @@ -684,7 +690,7 @@ new_dict_with_shared_keys(PyDictKeysObject *keys) for (i = 0; i < size; i++) { values->values[i] = NULL; } - return new_dict(keys, values); + return new_dict(keys, values, 0, 1); } @@ -733,7 +739,7 @@ PyObject * PyDict_New(void) { dictkeys_incref(Py_EMPTY_KEYS); - return new_dict(Py_EMPTY_KEYS, empty_values); + return new_dict(Py_EMPTY_KEYS, empty_values, 0, 0); } /* Search index of hash table from offset of entry table */ @@ -998,6 +1004,40 @@ insertion_resize(PyDictObject *mp) return dictresize(mp, calculate_log2_keysize(GROWTH_RATE(mp))); } +static int +insert_into_dictkeys(PyDictKeysObject *keys, PyObject *name) +{ + assert(PyUnicode_CheckExact(name)); + Py_hash_t hash = ((PyASCIIObject *)name)->hash; + if (hash == -1) { + hash = PyUnicode_Type.tp_hash(name); + if (hash == -1) { + PyErr_Clear(); + return DKIX_EMPTY; + } + } + Py_ssize_t ix = dictkeys_stringlookup(keys, name, hash); + if (ix == DKIX_EMPTY) { + if (keys->dk_usable <= 0) { + return DKIX_EMPTY; + } + Py_INCREF(name); + /* Insert into new slot. */ + keys->dk_version = 0; + Py_ssize_t hashpos = find_empty_slot(keys, hash); + ix = keys->dk_nentries; + PyDictKeyEntry *ep = &DK_ENTRIES(keys)[ix]; + dictkeys_set_index(keys, hashpos, ix); + assert(ep->me_key == NULL); + ep->me_key = name; + ep->me_hash = hash; + keys->dk_usable--; + keys->dk_nentries++; + } + assert (ix < SHARED_KEYS_MAX_SIZE); + return (int)ix; +} + /* Internal routine to insert a new item into the table. Used both by the internal resize routine and by the public insert routine. @@ -1043,7 +1083,7 @@ insertdict(PyDictObject *mp, PyObject *key, Py_hash_t hash, PyObject *value) Py_ssize_t index = mp->ma_keys->dk_nentries; assert(index < SHARED_KEYS_MAX_SIZE); assert((mp->ma_values->mv_order >> 60) == 0); - mp->ma_values->mv_order = (mp->ma_values->mv_order)<<4 | index; + mp->ma_values->mv_order = ((mp->ma_values->mv_order)<<4) | index; assert (mp->ma_values->values[index] == NULL); mp->ma_values->values[index] = value; } @@ -1144,8 +1184,7 @@ actually be smaller than the old one. If a table is split (its keys and hashes are shared, its values are not), then the values are temporarily copied into the table, it is resized as a combined table, then the me_value slots in the old table are NULLed out. -After resizing a table is always combined, -but can be resplit by make_keys_shared(). +After resizing a table is always combined. */ static int dictresize(PyDictObject *mp, uint8_t log2_newsize) @@ -1186,19 +1225,16 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) if (oldvalues != NULL) { /* Convert split table into new combined table. * We must incref keys; we can transfer values. - * Note that values of split table is always dense. */ for (Py_ssize_t i = 0; i < numentries; i++) { - int index = oldvalues->mv_order >> ((numentries-1-i)*4) & 15; - assert(oldvalues->values[index] != NULL); + int index = get_index_from_order(mp, i); PyDictKeyEntry *ep = &oldentries[index]; - PyObject *key = ep->me_key; - Py_INCREF(key); - newentries[i].me_key = key; + assert(oldvalues->values[index] != NULL); + Py_INCREF(ep->me_key); + newentries[i].me_key = ep->me_key; newentries[i].me_hash = ep->me_hash; newentries[i].me_value = oldvalues->values[index]; } - dictkeys_decref(oldkeys); mp->ma_values = NULL; if (oldvalues != empty_values) { @@ -1241,69 +1277,8 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) build_indices(mp->ma_keys, newentries, numentries); mp->ma_keys->dk_usable -= numentries; mp->ma_keys->dk_nentries = numentries; - return 0; -} - -/* Returns NULL if unable to split table. - * A NULL return does not necessarily indicate an error */ -static PyDictKeysObject * -make_keys_shared(PyObject *op) -{ - Py_ssize_t i; - Py_ssize_t size; - PyDictObject *mp = (PyDictObject *)op; - - if (!PyDict_CheckExact(op)) - return NULL; - if (mp->ma_used > SHARED_KEYS_MAX_SIZE) { - return NULL; - } - if (!_PyDict_HasSplitTable(mp)) { - PyDictKeyEntry *ep0; - PyDictValues *values; - assert(mp->ma_keys->dk_refcnt == 1); - if (mp->ma_keys->dk_kind == DICT_KEYS_GENERAL) { - return NULL; - } - else if (mp->ma_used > mp->ma_keys->dk_nentries) { - /* Remove dummy keys */ - if (dictresize(mp, DK_LOG_SIZE(mp->ma_keys))) - return NULL; - } - assert(mp->ma_used == mp->ma_keys->dk_nentries); - /* Copy values into a new array */ - ep0 = DK_ENTRIES(mp->ma_keys); - size = USABLE_FRACTION(DK_SIZE(mp->ma_keys)); - values = new_values(size); - if (values == NULL) { - PyErr_SetString(PyExc_MemoryError, - "Not enough memory to allocate new values array"); - return NULL; - } - uint64_t order = 0; - for (i = 0; i < mp->ma_used; i++) { - order <<= 4; - order |= i; - assert(ep0[i].me_value != NULL); - values->values[i] = ep0[i].me_value; - ep0[i].me_value = NULL; - } - values->mv_order = order; - for (; i < size; i++) { - assert(ep0[i].me_value == NULL); - values->values[i] = NULL; - ep0[i].me_value = NULL; - } - if (mp->ma_keys->dk_nentries + mp->ma_keys->dk_usable > SHARED_KEYS_MAX_SIZE) { - assert(mp->ma_keys->dk_nentries <= SHARED_KEYS_MAX_SIZE); - mp->ma_keys->dk_usable = SHARED_KEYS_MAX_SIZE - mp->ma_keys->dk_nentries; - } - mp->ma_keys->dk_kind = DICT_KEYS_SPLIT; - mp->ma_values = values; - } - dictkeys_incref(mp->ma_keys); ASSERT_CONSISTENT(mp); - return mp->ma_keys; + return 0; } PyObject * @@ -1331,7 +1306,7 @@ _PyDict_NewPresized(Py_ssize_t minused) new_keys = new_keys_object(log2_newsize); if (new_keys == NULL) return NULL; - return new_dict(new_keys, NULL); + return new_dict(new_keys, NULL, 0, 0); } /* Note that, for historical reasons, PyDict_GetItem() suppresses all errors @@ -1503,6 +1478,9 @@ _PyDict_GetItemStringWithError(PyObject *v, const char *key) /* Fast version of global value lookup (LOAD_GLOBAL). * Lookup in globals, then builtins. * + * + * + * * Raise an exception and return NULL if an error occurred (ex: computing the * key hash failed, key comparison failed, ...). Return NULL if the key doesn't * exist. Return the value if the key exists. @@ -1590,6 +1568,21 @@ _PyDict_SetItem_KnownHash(PyObject *op, PyObject *key, PyObject *value, return insertdict(mp, key, hash, value); } +static uint64_t +delete_index_from_order(uint64_t order, Py_ssize_t ix) +{ /* Update order */ + for (int i = 0;; i+= 4) { + assert (i < 64); + if (((order >> i) & 15) == (uint64_t)ix) { + /* Remove 4 bits at ith position */ + uint64_t high = ((order>>i)>>4)<= 0); mp->ma_used--; - mp->ma_keys->dk_version = 0; mp->ma_version_tag = DICT_NEXT_VERSION(); ep = &DK_ENTRIES(mp->ma_keys)[ix]; if (mp->ma_values) { @@ -1609,19 +1601,12 @@ delitem_common(PyDictObject *mp, Py_hash_t hash, Py_ssize_t ix, mp->ma_values->values[ix] = NULL; assert(ix < SHARED_KEYS_MAX_SIZE); /* Update order */ - for (int i = 0;; i+= 4) { - assert (i < 64); - if (((mp->ma_values->mv_order >> i) & 15) == (uint64_t)ix) { - /* Remove 4 bits at ith position */ - uint64_t order = mp->ma_values->mv_order; - uint64_t high = ((order>>i)>>4)<ma_values->mv_order = high | low; - break; - } - } + mp->ma_values->mv_order = + delete_index_from_order(mp->ma_values->mv_order, ix); + ASSERT_CONSISTENT(mp); } else { + mp->ma_keys->dk_version = 0; dictkeys_set_index(mp->ma_keys, hashpos, DKIX_DUMMY); old_key = ep->me_key; ep->me_key = NULL; @@ -2692,7 +2677,7 @@ PyDict_Copy(PyObject *o) if (_PyDict_HasSplitTable(mp)) { PyDictObject *split_copy; - Py_ssize_t size = USABLE_FRACTION(DK_SIZE(mp->ma_keys)); + Py_ssize_t size = shared_keys_usable_size(mp->ma_keys); PyDictValues *newvalues; newvalues = new_values(size); if (newvalues == NULL) @@ -2740,7 +2725,7 @@ PyDict_Copy(PyObject *o) if (keys == NULL) { return NULL; } - PyDictObject *new = (PyDictObject *)new_dict(keys, NULL); + PyDictObject *new = (PyDictObject *)new_dict(keys, NULL, 0, 0); if (new == NULL) { /* In case of an error, `new_dict()` takes care of cleaning up `keys`. */ @@ -2979,15 +2964,6 @@ PyDict_SetDefault(PyObject *d, PyObject *key, PyObject *defaultobj) if (ix == DKIX_ERROR) return NULL; - if (_PyDict_HasSplitTable(mp) && - ((ix >= 0 && value == NULL && mp->ma_used != ix) || - (ix == DKIX_EMPTY && mp->ma_used != mp->ma_keys->dk_nentries))) { - if (insertion_resize(mp) < 0) { - return NULL; - } - ix = DKIX_EMPTY; - } - if (ix == DKIX_EMPTY) { mp->ma_keys->dk_version = 0; PyDictKeyEntry *ep, *ep0; @@ -3028,7 +3004,7 @@ PyDict_SetDefault(PyObject *d, PyObject *key, PyObject *defaultobj) else if (value == NULL) { value = defaultobj; assert(_PyDict_HasSplitTable(mp)); - assert(ix == mp->ma_used); + assert(mp->ma_values->values[ix] == NULL); Py_INCREF(value); MAINTAIN_TRACKING(mp, key, value); mp->ma_values->values[ix] = value; @@ -3204,20 +3180,22 @@ static PyObject *dictiter_new(PyDictObject *, PyTypeObject *); Py_ssize_t _PyDict_SizeOf(PyDictObject *mp) { - Py_ssize_t size, usable, res; + Py_ssize_t size, res; size = DK_SIZE(mp->ma_keys); - usable = USABLE_FRACTION(size); res = _PyObject_SIZE(Py_TYPE(mp)); - if (mp->ma_values) - res += usable * sizeof(PyObject*); + if (mp->ma_values) { + res += shared_keys_usable_size(mp->ma_keys) * sizeof(PyObject*); + } /* If the dictionary is split, the keys portion is accounted-for in the type object. */ - if (mp->ma_keys->dk_refcnt == 1) + if (mp->ma_keys->dk_refcnt == 1) { + Py_ssize_t usable = USABLE_FRACTION(size); res += (sizeof(PyDictKeysObject) + DK_IXSIZE(mp->ma_keys) * size + sizeof(PyDictKeyEntry) * usable); + } return res; } @@ -4919,11 +4897,14 @@ dictvalues_reversed(_PyDictViewObject *dv, PyObject *Py_UNUSED(ignored)) PyDictKeysObject * _PyDict_NewKeysForClass(void) { - PyDictKeysObject *keys = new_keys_object(PyDict_LOG_MINSIZE); + PyDictKeysObject *keys = new_keys_object(5); /* log2(32) */ if (keys == NULL) { PyErr_Clear(); } else { + assert(keys->dk_nentries == 0); + /* Set to max size+1 as it will shrink by one before each new object */ + keys->dk_usable = SHARED_KEYS_MAX_SIZE; keys->dk_kind = DICT_KEYS_SPLIT; } return keys; @@ -4931,15 +4912,42 @@ _PyDict_NewKeysForClass(void) #define CACHED_KEYS(tp) (((PyHeapTypeObject*)tp)->ht_cached_keys) +static int +init_inline_values(PyObject *obj, PyTypeObject *tp) +{ + assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE); + assert(tp->tp_dictoffset > 0); + assert(tp->tp_inline_values_offset > 0); + PyDictKeysObject *keys = CACHED_KEYS(tp); + assert(keys != NULL); + if (keys->dk_usable > 1) { + keys->dk_usable--; + } + Py_ssize_t size = shared_keys_usable_size(keys); + assert(size > 0); + PyDictValues *values = new_values(size); + if (values == NULL) { + PyErr_NoMemory(); + return -1; + } + values->mv_order = 0; + for (int i = 0; i < size; i++) { + values->values[i] = NULL; + } + *((PyDictValues **)((char *)obj + tp->tp_inline_values_offset)) = values; + return 0; +} + int _PyObject_InitializeDict(PyObject *obj) { - PyObject **dictptr = _PyObject_GetDictPtr(obj); - if (dictptr == NULL) { + PyTypeObject *tp = Py_TYPE(obj); + if (tp->tp_dictoffset == 0) { return 0; } - assert(*dictptr == NULL); - PyTypeObject *tp = Py_TYPE(obj); + if (tp->tp_inline_values_offset) { + return init_inline_values(obj, tp); + } PyObject *dict; if (_PyType_HasFeature(tp, Py_TPFLAGS_HEAPTYPE) && CACHED_KEYS(tp)) { dictkeys_incref(CACHED_KEYS(tp)); @@ -4951,15 +4959,174 @@ _PyObject_InitializeDict(PyObject *obj) if (dict == NULL) { return -1; } + PyObject **dictptr = _PyObject_DictPointer(obj); *dictptr = dict; return 0; } +static PyObject * +make_dict_from_instance_attributes(PyDictKeysObject *keys, PyDictValues *values) +{ + dictkeys_incref(keys); + Py_ssize_t used = 0; + Py_ssize_t track = 0; + for (Py_ssize_t i = 0; i < shared_keys_usable_size(keys); i++) { + PyObject *val = values->values[i]; + if (val != NULL) { + used += 1; + track += _PyObject_GC_MAY_BE_TRACKED(val); + } + } + PyObject *res = new_dict(keys, values, used, 0); + if (track && res) { + _PyObject_GC_TRACK(res); + } + return res; +} + +PyObject * +_PyObject_MakeDictFromInstanceAttributes(PyObject *obj, PyDictValues *values) +{ + assert(Py_TYPE(obj)->tp_inline_values_offset != 0); + PyDictKeysObject *keys = CACHED_KEYS(Py_TYPE(obj)); + return make_dict_from_instance_attributes(keys, values); +} + +int +_PyObject_StoreInstanceAttribute(PyObject *obj, PyDictValues *values, + PyObject *name, PyObject *value) +{ + assert(PyUnicode_CheckExact(name)); + PyTypeObject *tp = Py_TYPE(obj); + PyDictKeysObject *keys = CACHED_KEYS(Py_TYPE(obj)); + assert(keys != NULL); + assert(values != NULL); + int ix = insert_into_dictkeys(keys, name); + if (ix == DKIX_EMPTY) { + if (value == NULL) { + PyErr_SetObject(PyExc_AttributeError, name); + return -1; + } + PyObject *dict = make_dict_from_instance_attributes(keys, values); + if (dict == NULL) { + return -1; + } + *((PyDictValues **)((char *)obj + tp->tp_inline_values_offset)) = NULL; + *((PyObject **) ((char *)obj + tp->tp_dictoffset)) = dict; + return PyDict_SetItem(dict, name, value); + } + PyObject *old_value = values->values[ix]; + Py_XINCREF(value); + values->values[ix] = value; + if (old_value == NULL) { + if (value == NULL) { + PyErr_SetObject(PyExc_AttributeError, name); + return -1; + } + values->mv_order = (values->mv_order << 4) | ix; + } + else { + if (value == NULL) { + values->mv_order = delete_index_from_order(values->mv_order, ix); + } + Py_DECREF(old_value); + } + return 0; +} + +PyObject * +_PyObject_GetInstanceAttribute(PyObject *obj, PyDictValues *values, + PyObject *name) +{ + assert(PyUnicode_CheckExact(name)); + PyDictKeysObject *keys = CACHED_KEYS(Py_TYPE(obj)); + assert(keys != NULL); + Py_ssize_t ix = _PyDictKeys_StringLookup(keys, name); + if (ix == DKIX_EMPTY) { + return NULL; + } + PyObject *value = values->values[ix]; + Py_XINCREF(value); + return value; +} + +int +_PyObject_IsInstanceDictEmpty(PyObject *obj) +{ + PyTypeObject *tp = Py_TYPE(obj); + if (tp->tp_dictoffset == 0) { + return 1; + } + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr && *values_ptr) { + PyDictKeysObject *keys = CACHED_KEYS(tp); + for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) { + if ((*values_ptr)->values[i] != NULL) { + return 0; + } + } + return 1; + } + PyObject **dictptr = _PyObject_DictPointer(obj); + PyObject *dict = *dictptr; + if (dict == NULL) { + return 1; + } + return ((PyDictObject *)dict)->ma_used == 0; +} + + +int +_PyObject_VisitInstanceAttributes(PyObject *self, visitproc visit, void *arg) +{ + PyTypeObject *tp = Py_TYPE(self); + assert(tp->tp_inline_values_offset); + PyDictValues **values_ptr = _PyObject_ValuesPointer(self); + if (*values_ptr == NULL) { + return 0; + } + PyDictKeysObject *keys = CACHED_KEYS(tp); + for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) { + Py_VISIT((*values_ptr)->values[i]); + } + return 0; +} + +void +_PyObject_ClearInstanceAttributes(PyObject *self) +{ + PyTypeObject *tp = Py_TYPE(self); + assert(tp->tp_inline_values_offset); + PyDictValues **values_ptr = _PyObject_ValuesPointer(self); + if (*values_ptr == NULL) { + return; + } + PyDictKeysObject *keys = CACHED_KEYS(tp); + for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) { + Py_CLEAR((*values_ptr)->values[i]); + } +} + +void +_PyObject_FreeInstanceAttributes(PyObject *self) +{ + PyTypeObject *tp = Py_TYPE(self); + assert(tp->tp_inline_values_offset); + PyDictValues **values_ptr = _PyObject_ValuesPointer(self); + if (*values_ptr == NULL) { + return; + } + PyDictKeysObject *keys = CACHED_KEYS(tp); + for (Py_ssize_t i = 0; i < keys->dk_nentries; i++) { + Py_XDECREF((*values_ptr)->values[i]); + } + free_values(*values_ptr); +} PyObject * PyObject_GenericGetDict(PyObject *obj, void *context) { - PyObject **dictptr = _PyObject_GetDictPtr(obj); + PyObject **dictptr = _PyObject_DictPointer(obj); if (dictptr == NULL) { PyErr_SetString(PyExc_AttributeError, "This object has no __dict__"); @@ -4968,7 +5135,14 @@ PyObject_GenericGetDict(PyObject *obj, void *context) PyObject *dict = *dictptr; if (dict == NULL) { PyTypeObject *tp = Py_TYPE(obj); - if (_PyType_HasFeature(tp, Py_TPFLAGS_HEAPTYPE) && CACHED_KEYS(tp)) { + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr && *values_ptr) { + *dictptr = dict = make_dict_from_instance_attributes(CACHED_KEYS(tp), *values_ptr); + if (dict != NULL) { + *values_ptr = NULL; + } + } + else if (_PyType_HasFeature(tp, Py_TPFLAGS_HEAPTYPE) && CACHED_KEYS(tp)) { dictkeys_incref(CACHED_KEYS(tp)); *dictptr = dict = new_dict_with_shared_keys(CACHED_KEYS(tp)); } @@ -5003,37 +5177,7 @@ _PyObjectDict_SetItem(PyTypeObject *tp, PyObject **dictptr, res = PyDict_DelItem(dict, key); } else { - int was_shared = (cached == ((PyDictObject *)dict)->ma_keys); res = PyDict_SetItem(dict, key, value); - if (was_shared && - (cached = CACHED_KEYS(tp)) != NULL && - cached != ((PyDictObject *)dict)->ma_keys && - cached->dk_nentries <= SHARED_KEYS_MAX_SIZE) { - /* PyDict_SetItem() may call dictresize and convert split table - * into combined table. In such case, convert it to split - * table again and update type's shared key only when this is - * the only dict sharing key with the type. - * - * This is to allow using shared key in class like this: - * - * class C: - * def __init__(self): - * # one dict resize happens - * self.a, self.b, self.c = 1, 2, 3 - * self.d, self.e, self.f = 4, 5, 6 - * a = C() - */ - if (cached->dk_refcnt == 1) { - PyDictKeysObject *new_cached = make_keys_shared(dict); - if (new_cached != NULL) { - CACHED_KEYS(tp) = new_cached; - dictkeys_decref(cached); - } - else if (PyErr_Occurred()) { - return -1; - } - } - } } } else { dict = *dictptr; diff --git a/Objects/object.c b/Objects/object.c index 9d48346d6d227..14c85c233df11 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -5,6 +5,7 @@ #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _Py_EnterRecursiveCall() #include "pycore_context.h" +#include "pycore_dict.h" #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() #include "pycore_object.h" // _PyType_CheckConsistency() #include "pycore_pyerrors.h" // _PyErr_Occurred() @@ -1065,10 +1066,8 @@ PyObject_SetAttr(PyObject *v, PyObject *name, PyObject *value) return -1; } -/* Helper to get a pointer to an object's __dict__ slot, if any */ - PyObject ** -_PyObject_GetDictPtr(PyObject *obj) +_PyObject_DictPointer(PyObject *obj) { Py_ssize_t dictoffset; PyTypeObject *tp = Py_TYPE(obj); @@ -1090,6 +1089,35 @@ _PyObject_GetDictPtr(PyObject *obj) return (PyObject **) ((char *)obj + dictoffset); } +/* Helper to get a pointer to an object's __dict__ slot, if any. + * Creates the dict from inline attributes if necessary. + * Does not set an exception. */ +PyObject ** +_PyObject_GetDictPtr(PyObject *obj) +{ + PyObject **dict_ptr = _PyObject_DictPointer(obj); + if (dict_ptr == NULL) { + return NULL; + } + if (*dict_ptr != NULL) { + return dict_ptr; + } + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr == NULL || *values_ptr == NULL) { + return dict_ptr; + } + PyObject *dict = _PyObject_MakeDictFromInstanceAttributes(obj, *values_ptr); + if (dict == NULL) { + PyErr_Clear(); + return NULL; + } + assert(*dict_ptr == NULL); + assert(*values_ptr != NULL); + *values_ptr = NULL; + *dict_ptr = dict; + return dict_ptr; +} + PyObject * PyObject_SelfIter(PyObject *obj) { @@ -1136,7 +1164,7 @@ _PyObject_GetMethod(PyObject *obj, PyObject *name, PyObject **method) } } - if (tp->tp_getattro != PyObject_GenericGetAttr || !PyUnicode_Check(name)) { + if (tp->tp_getattro != PyObject_GenericGetAttr || !PyUnicode_CheckExact(name)) { *method = PyObject_GetAttr(obj, name); return 0; } @@ -1156,23 +1184,34 @@ _PyObject_GetMethod(PyObject *obj, PyObject *name, PyObject **method) } } } - - PyObject **dictptr = _PyObject_GetDictPtr(obj); - PyObject *dict; - if (dictptr != NULL && (dict = *dictptr) != NULL) { - Py_INCREF(dict); - PyObject *attr = PyDict_GetItemWithError(dict, name); + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr && *values_ptr) { + assert(*_PyObject_DictPointer(obj) == NULL); + PyObject *attr = _PyObject_GetInstanceAttribute(obj, *values_ptr, name); if (attr != NULL) { - *method = Py_NewRef(attr); - Py_DECREF(dict); + *method = attr; Py_XDECREF(descr); return 0; } - Py_DECREF(dict); + } + else { + PyObject **dictptr = _PyObject_DictPointer(obj); + PyObject *dict; + if (dictptr != NULL && (dict = *dictptr) != NULL) { + Py_INCREF(dict); + PyObject *attr = PyDict_GetItemWithError(dict, name); + if (attr != NULL) { + *method = Py_NewRef(attr); + Py_DECREF(dict); + Py_XDECREF(descr); + return 0; + } + Py_DECREF(dict); - if (PyErr_Occurred()) { - Py_XDECREF(descr); - return 0; + if (PyErr_Occurred()) { + Py_XDECREF(descr); + return 0; + } } } @@ -1200,6 +1239,17 @@ _PyObject_GetMethod(PyObject *obj, PyObject *name, PyObject **method) return 0; } +PyDictValues ** +_PyObject_ValuesPointer(PyObject *obj) +{ + PyTypeObject *tp = Py_TYPE(obj); + Py_ssize_t offset = tp->tp_inline_values_offset; + if (offset == 0) { + return NULL; + } + return (PyDictValues **) ((char *)obj + offset); +} + /* Generic GetAttr functions - put these in your tp_[gs]etattro slot. */ PyObject * @@ -1247,25 +1297,46 @@ _PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name, goto done; } } - if (dict == NULL) { - /* Inline _PyObject_GetDictPtr */ - dictoffset = tp->tp_dictoffset; - if (dictoffset != 0) { - if (dictoffset < 0) { - Py_ssize_t tsize = Py_SIZE(obj); - if (tsize < 0) { - tsize = -tsize; + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr && *values_ptr) { + if (PyUnicode_CheckExact(name)) { + assert(*_PyObject_DictPointer(obj) == NULL); + res = _PyObject_GetInstanceAttribute(obj, *values_ptr, name); + if (res != NULL) { + goto done; } - size_t size = _PyObject_VAR_SIZE(tp, tsize); - _PyObject_ASSERT(obj, size <= PY_SSIZE_T_MAX); - - dictoffset += (Py_ssize_t)size; - _PyObject_ASSERT(obj, dictoffset > 0); - _PyObject_ASSERT(obj, dictoffset % SIZEOF_VOID_P == 0); } - dictptr = (PyObject **) ((char *)obj + dictoffset); - dict = *dictptr; + else { + dictptr = _PyObject_DictPointer(obj); + assert(dictptr != NULL && *dictptr == NULL); + *dictptr = dict = _PyObject_MakeDictFromInstanceAttributes(obj, *values_ptr); + if (dict == NULL) { + res = NULL; + goto done; + } + *values_ptr = NULL; + } + } + else { + /* Inline _PyObject_DictPointer */ + dictoffset = tp->tp_dictoffset; + if (dictoffset != 0) { + if (dictoffset < 0) { + Py_ssize_t tsize = Py_SIZE(obj); + if (tsize < 0) { + tsize = -tsize; + } + size_t size = _PyObject_VAR_SIZE(tp, tsize); + _PyObject_ASSERT(obj, size <= PY_SSIZE_T_MAX); + + dictoffset += (Py_ssize_t)size; + _PyObject_ASSERT(obj, dictoffset > 0); + _PyObject_ASSERT(obj, dictoffset % SIZEOF_VOID_P == 0); + } + dictptr = (PyObject **) ((char *)obj + dictoffset); + dict = *dictptr; + } } } if (dict != NULL) { @@ -1328,7 +1399,6 @@ _PyObject_GenericSetAttrWithDict(PyObject *obj, PyObject *name, PyTypeObject *tp = Py_TYPE(obj); PyObject *descr; descrsetfunc f; - PyObject **dictptr; int res = -1; if (!PyUnicode_Check(name)){ @@ -1354,30 +1424,30 @@ _PyObject_GenericSetAttrWithDict(PyObject *obj, PyObject *name, } } - /* XXX [Steve Dower] These are really noisy - worth it? */ - /*if (PyType_Check(obj) || PyModule_Check(obj)) { - if (value && PySys_Audit("object.__setattr__", "OOO", obj, name, value) < 0) - return -1; - if (!value && PySys_Audit("object.__delattr__", "OO", obj, name) < 0) - return -1; - }*/ - if (dict == NULL) { - dictptr = _PyObject_GetDictPtr(obj); - if (dictptr == NULL) { - if (descr == NULL) { - PyErr_Format(PyExc_AttributeError, - "'%.100s' object has no attribute '%U'", - tp->tp_name, name); + PyDictValues **values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr && *values_ptr) { + res = _PyObject_StoreInstanceAttribute(obj, *values_ptr, name, value); + } + else { + PyObject **dictptr = _PyObject_DictPointer(obj); + if (dictptr == NULL) { + if (descr == NULL) { + PyErr_Format(PyExc_AttributeError, + "'%.100s' object has no attribute '%U'", + tp->tp_name, name); + } + else { + PyErr_Format(PyExc_AttributeError, + "'%.50s' object attribute '%U' is read-only", + tp->tp_name, name); + } + goto done; } else { - PyErr_Format(PyExc_AttributeError, - "'%.50s' object attribute '%U' is read-only", - tp->tp_name, name); + res = _PyObjectDict_SetItem(tp, dictptr, name, value); } - goto done; } - res = _PyObjectDict_SetItem(tp, dictptr, name, value); } else { Py_INCREF(dict); @@ -1407,8 +1477,15 @@ PyObject_GenericSetDict(PyObject *obj, PyObject *value, void *context) { PyObject **dictptr = _PyObject_GetDictPtr(obj); if (dictptr == NULL) { - PyErr_SetString(PyExc_AttributeError, - "This object has no __dict__"); + PyDictValues** values_ptr = _PyObject_ValuesPointer(obj); + if (values_ptr != NULL && *values_ptr != NULL) { + /* Was unable to convert to dict */ + PyErr_NoMemory(); + } + else { + PyErr_SetString(PyExc_AttributeError, + "This object has no __dict__"); + } return -1; } if (value == NULL) { diff --git a/Objects/typeobject.c b/Objects/typeobject.c index 544e8a4aa2211..aa0733361e58d 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -1232,8 +1232,16 @@ subtype_traverse(PyObject *self, visitproc visit, void *arg) assert(base); } + if (type->tp_inline_values_offset) { + assert(type->tp_dictoffset); + int err = _PyObject_VisitInstanceAttributes(self, visit, arg); + if (err) { + return err; + } + } + if (type->tp_dictoffset != base->tp_dictoffset) { - PyObject **dictptr = _PyObject_GetDictPtr(self); + PyObject **dictptr = _PyObject_DictPointer(self); if (dictptr && *dictptr) Py_VISIT(*dictptr); } @@ -1293,8 +1301,11 @@ subtype_clear(PyObject *self) /* Clear the instance dict (if any), to break cycles involving only __dict__ slots (as in the case 'self.__dict__ is self'). */ + if (type->tp_inline_values_offset) { + _PyObject_ClearInstanceAttributes(self); + } if (type->tp_dictoffset != base->tp_dictoffset) { - PyObject **dictptr = _PyObject_GetDictPtr(self); + PyObject **dictptr = _PyObject_DictPointer(self); if (dictptr && *dictptr) Py_CLEAR(*dictptr); } @@ -1433,9 +1444,12 @@ subtype_dealloc(PyObject *self) assert(base); } - /* If we added a dict, DECREF it */ + /* If we added a dict, DECREF it, or free inline values. */ + if (type->tp_inline_values_offset) { + _PyObject_FreeInstanceAttributes(self); + } if (type->tp_dictoffset && !base->tp_dictoffset) { - PyObject **dictptr = _PyObject_GetDictPtr(self); + PyObject **dictptr = _PyObject_DictPointer(self); if (dictptr != NULL) { PyObject *dict = *dictptr; if (dict != NULL) { @@ -2159,7 +2173,6 @@ mro_internal(PyTypeObject *type, PyObject **p_old_mro) return 1; } - /* Calculate the best base amongst multiple base classes. This is the first one that's on the path to the "solid base". */ @@ -2230,6 +2243,10 @@ extra_ivars(PyTypeObject *type, PyTypeObject *base) return t_size != b_size || type->tp_itemsize != base->tp_itemsize; } + if (type->tp_inline_values_offset && base->tp_inline_values_offset == 0 && + type->tp_inline_values_offset + sizeof(PyDictValues *) == t_size && + type->tp_flags & Py_TPFLAGS_HEAPTYPE) + t_size -= sizeof(PyDictValues *); if (type->tp_weaklistoffset && base->tp_weaklistoffset == 0 && type->tp_weaklistoffset + sizeof(PyObject *) == t_size && type->tp_flags & Py_TPFLAGS_HEAPTYPE) @@ -2238,7 +2255,6 @@ extra_ivars(PyTypeObject *type, PyTypeObject *base) type->tp_dictoffset + sizeof(PyObject *) == t_size && type->tp_flags & Py_TPFLAGS_HEAPTYPE) t_size -= sizeof(PyObject *); - return t_size != b_size; } @@ -2258,6 +2274,7 @@ solid_base(PyTypeObject *type) } static void object_dealloc(PyObject *); +static PyObject *object_new(PyTypeObject *, PyObject *, PyObject *); static int object_init(PyObject *, PyObject *, PyObject *); static int update_slot(PyTypeObject *, PyObject *); static void fixup_slot_dispatchers(PyTypeObject *); @@ -2979,6 +2996,13 @@ type_new_descriptors(const type_new_ctx *ctx, PyTypeObject *type) type->tp_weaklistoffset = slotoffset; slotoffset += sizeof(PyObject *); } + if (type->tp_dictoffset > 0) { + type->tp_inline_values_offset = slotoffset; + slotoffset += sizeof(PyDictValues *); + } + else { + type->tp_inline_values_offset = 0; + } type->tp_basicsize = slotoffset; type->tp_itemsize = ctx->base->tp_itemsize; @@ -3181,7 +3205,8 @@ type_new_impl(type_new_ctx *ctx) // Put the proper slots in place fixup_slot_dispatchers(type); - if (type->tp_dictoffset) { + if (type->tp_inline_values_offset) { + assert(type->tp_dictoffset > 0); PyHeapTypeObject *et = (PyHeapTypeObject*)type; et->ht_cached_keys = _PyDict_NewKeysForClass(); } @@ -3195,6 +3220,7 @@ type_new_impl(type_new_ctx *ctx) } assert(_PyType_CheckConsistency(type)); + return (PyObject *)type; error: @@ -3550,7 +3576,8 @@ PyType_FromModuleAndSpec(PyObject *module, PyType_Spec *spec, PyObject *bases) if (PyType_Ready(type) < 0) goto fail; - if (type->tp_dictoffset) { + if (type->tp_inline_values_offset) { + assert(type->tp_dictoffset > 0); res->ht_cached_keys = _PyDict_NewKeysForClass(); } @@ -4257,7 +4284,6 @@ type_traverse(PyTypeObject *type, visitproc visit, void *arg) static int type_clear(PyTypeObject *type) { - PyDictKeysObject *cached_keys; /* Because of type_is_gc(), the collector only calls this for heaptypes. */ _PyObject_ASSERT((PyObject *)type, type->tp_flags & Py_TPFLAGS_HEAPTYPE); @@ -4292,11 +4318,6 @@ type_clear(PyTypeObject *type) */ PyType_Modified(type); - cached_keys = ((PyHeapTypeObject *)type)->ht_cached_keys; - if (cached_keys != NULL) { - ((PyHeapTypeObject *)type)->ht_cached_keys = NULL; - _PyDictKeys_DecRef(cached_keys); - } if (type->tp_dict) { PyDict_Clear(type->tp_dict); } @@ -4618,6 +4639,7 @@ compatible_with_tp_base(PyTypeObject *child) child->tp_itemsize == parent->tp_itemsize && child->tp_dictoffset == parent->tp_dictoffset && child->tp_weaklistoffset == parent->tp_weaklistoffset && + child->tp_inline_values_offset == parent->tp_inline_values_offset && ((child->tp_flags & Py_TPFLAGS_HAVE_GC) == (parent->tp_flags & Py_TPFLAGS_HAVE_GC)) && (child->tp_dealloc == subtype_dealloc || @@ -4637,6 +4659,8 @@ same_slots_added(PyTypeObject *a, PyTypeObject *b) size += sizeof(PyObject *); if (a->tp_weaklistoffset == size && b->tp_weaklistoffset == size) size += sizeof(PyObject *); + if (a->tp_inline_values_offset == size && b->tp_inline_values_offset == size) + size += sizeof(PyObject *); /* Check slots compliance */ if (!(a->tp_flags & Py_TPFLAGS_HEAPTYPE) || @@ -4781,6 +4805,17 @@ object_set_class(PyObject *self, PyObject *value, void *closure) } if (compatible_for_assignment(oldto, newto, "__class__")) { + /* Changing the class will change the implicit dict keys, + * so we must materialize the dictionary first. */ + assert(oldto->tp_inline_values_offset == newto->tp_inline_values_offset); + _PyObject_GetDictPtr(self); + PyDictValues** values_ptr = _PyObject_ValuesPointer(self); + if (values_ptr != NULL && *values_ptr != NULL) { + /* Was unable to convert to dict */ + PyErr_NoMemory(); + return -1; + } + assert(_PyObject_ValuesPointer(self) == NULL || *_PyObject_ValuesPointer(self) == NULL); if (newto->tp_flags & Py_TPFLAGS_HEAPTYPE) { Py_INCREF(newto); } @@ -4906,23 +4941,16 @@ _PyObject_GetState(PyObject *obj, int required) Py_TYPE(obj)->tp_name); return NULL; } - - { - PyObject **dict; - dict = _PyObject_GetDictPtr(obj); - /* It is possible that the object's dict is not initialized - yet. In this case, we will return None for the state. - We also return None if the dict is empty to make the behavior - consistent regardless whether the dict was initialized or not. - This make unit testing easier. */ - if (dict != NULL && *dict != NULL && PyDict_GET_SIZE(*dict)) { - state = *dict; - } - else { - state = Py_None; - } + if (_PyObject_IsInstanceDictEmpty(obj)) { + state = Py_None; Py_INCREF(state); } + else { + state = PyObject_GenericGetDict(obj, NULL); + if (state == NULL) { + return NULL; + } + } slotnames = _PyType_GetSlotNames(Py_TYPE(obj)); if (slotnames == NULL) { @@ -4933,12 +4961,18 @@ _PyObject_GetState(PyObject *obj, int required) assert(slotnames == Py_None || PyList_Check(slotnames)); if (required) { Py_ssize_t basicsize = PyBaseObject_Type.tp_basicsize; - if (Py_TYPE(obj)->tp_dictoffset) + if (Py_TYPE(obj)->tp_dictoffset) { basicsize += sizeof(PyObject *); - if (Py_TYPE(obj)->tp_weaklistoffset) + } + if (Py_TYPE(obj)->tp_weaklistoffset) { basicsize += sizeof(PyObject *); - if (slotnames != Py_None) + } + if (Py_TYPE(obj)->tp_inline_values_offset) { + basicsize += sizeof(PyDictValues *); + } + if (slotnames != Py_None) { basicsize += sizeof(PyObject *) * PyList_GET_SIZE(slotnames); + } if (Py_TYPE(obj)->tp_basicsize > basicsize) { Py_DECREF(slotnames); Py_DECREF(state); @@ -5708,6 +5742,7 @@ inherit_special(PyTypeObject *type, PyTypeObject *base) COPYVAL(tp_itemsize); COPYVAL(tp_weaklistoffset); COPYVAL(tp_dictoffset); + COPYVAL(tp_inline_values_offset); #undef COPYVAL /* Setup fast subclass flags */ diff --git a/Python/ceval.c b/Python/ceval.c index f4cacd84b66cb..5cf2ab341ddc4 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -3618,7 +3618,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(LOAD_ATTR_SPLIT_KEYS) { + TARGET(LOAD_ATTR_INSTANCE_VALUE) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyObject *res; @@ -3629,11 +3629,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(cache1->tp_version != 0); DEOPT_IF(tp->tp_version_tag != cache1->tp_version, LOAD_ATTR); assert(tp->tp_dictoffset > 0); - PyDictObject *dict = *(PyDictObject **)(((char *)owner) + tp->tp_dictoffset); - DEOPT_IF(dict == NULL, LOAD_ATTR); - assert(PyDict_CheckExact((PyObject *)dict)); - DEOPT_IF(dict->ma_keys->dk_version != cache1->dk_version_or_hint, LOAD_ATTR); - res = dict->ma_values->values[cache0->index]; + assert(tp->tp_inline_values_offset > 0); + PyDictValues *values = *(PyDictValues **)(((char *)owner) + tp->tp_inline_values_offset); + DEOPT_IF(values == NULL, LOAD_ATTR); + res = values->values[cache0->index]; DEOPT_IF(res == NULL, LOAD_ATTR); STAT_INC(LOAD_ATTR, hit); record_cache_hit(cache0); @@ -3725,7 +3724,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } } - TARGET(STORE_ATTR_SPLIT_KEYS) { + TARGET(STORE_ATTR_INSTANCE_VALUE) { assert(cframe.use_tracing == 0); PyObject *owner = TOP(); PyTypeObject *tp = Py_TYPE(owner); @@ -3735,31 +3734,23 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(cache1->tp_version != 0); DEOPT_IF(tp->tp_version_tag != cache1->tp_version, STORE_ATTR); assert(tp->tp_dictoffset > 0); - PyDictObject *dict = *(PyDictObject **)(((char *)owner) + tp->tp_dictoffset); - DEOPT_IF(dict == NULL, STORE_ATTR); - assert(PyDict_CheckExact((PyObject *)dict)); - DEOPT_IF(dict->ma_keys->dk_version != cache1->dk_version_or_hint, STORE_ATTR); + assert(tp->tp_inline_values_offset > 0); + PyDictValues *values = *(PyDictValues **)(((char *)owner) + tp->tp_inline_values_offset); + DEOPT_IF(values == NULL, STORE_ATTR); STAT_INC(STORE_ATTR, hit); record_cache_hit(cache0); int index = cache0->index; STACK_SHRINK(1); PyObject *value = POP(); - PyObject *old_value = dict->ma_values->values[index]; - dict->ma_values->values[index] = value; + PyObject *old_value = values->values[index]; + values->values[index] = value; if (old_value == NULL) { assert(index < 16); - dict->ma_values->mv_order = (dict->ma_values->mv_order << 4) | index; - dict->ma_used++; + values->mv_order = (values->mv_order << 4) | index; } else { Py_DECREF(old_value); } - /* Ensure dict is GC tracked if it needs to be */ - if (!_PyObject_GC_IS_TRACKED(dict) && _PyObject_GC_MAY_BE_TRACKED(value)) { - _PyObject_GC_TRACK(dict); - } - /* PEP 509 */ - dict->ma_version_tag = DICT_NEXT_VERSION(); Py_DECREF(owner); DISPATCH(); } @@ -4474,21 +4465,31 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr _PyObjectCache *cache2 = &caches[-2].obj; DEOPT_IF(self_cls->tp_version_tag != cache1->tp_version, LOAD_METHOD); - assert(cache1->dk_version_or_hint != 0); - assert(cache1->tp_version != 0); - assert(self_cls->tp_dictoffset >= 0); - assert(Py_TYPE(self_cls)->tp_dictoffset > 0); - - // inline version of _PyObject_GetDictPtr for offset >= 0 - PyObject *dict = self_cls->tp_dictoffset != 0 ? - *(PyObject **) ((char *)self + self_cls->tp_dictoffset) : NULL; - - // Ensure self.__dict__ didn't modify keys. - // Don't care if self has no dict, it could be builtin or __slots__. - DEOPT_IF(dict != NULL && - ((PyDictObject *)dict)->ma_keys->dk_version != - cache1->dk_version_or_hint, LOAD_METHOD); + assert(self_cls->tp_dictoffset > 0); + assert(self_cls->tp_inline_values_offset > 0); + PyDictObject *dict = *(PyDictObject **)(((char *)self) + self_cls->tp_dictoffset); + DEOPT_IF(dict != NULL, LOAD_METHOD); + DEOPT_IF(((PyHeapTypeObject *)self_cls)->ht_cached_keys->dk_version != cache1->dk_version_or_hint, LOAD_METHOD); + STAT_INC(LOAD_METHOD, hit); + record_cache_hit(cache0); + PyObject *res = cache2->obj; + assert(res != NULL); + assert(_PyType_HasFeature(Py_TYPE(res), Py_TPFLAGS_METHOD_DESCRIPTOR)); + Py_INCREF(res); + SET_TOP(res); + PUSH(self); + DISPATCH(); + } + TARGET(LOAD_METHOD_NO_DICT) { + PyObject *self = TOP(); + PyTypeObject *self_cls = Py_TYPE(self); + SpecializedCacheEntry *caches = GET_CACHE(); + _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + _PyAttrCache *cache1 = &caches[-1].attr; + _PyObjectCache *cache2 = &caches[-2].obj; + DEOPT_IF(self_cls->tp_version_tag != cache1->tp_version, LOAD_METHOD); + assert(self_cls->tp_dictoffset == 0); STAT_INC(LOAD_METHOD, hit); record_cache_hit(cache0); PyObject *res = cache2->obj; @@ -4530,7 +4531,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr record_cache_hit(cache0); PyObject *res = cache2->obj; assert(res != NULL); - assert(_PyType_HasFeature(Py_TYPE(res), Py_TPFLAGS_METHOD_DESCRIPTOR)); Py_INCREF(res); SET_TOP(NULL); Py_DECREF(cls); diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 30df68382536a..773f9254cf624 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -41,7 +41,7 @@ static void *opcode_targets[256] = { &&TARGET_BINARY_SUBSCR_DICT, &&TARGET_JUMP_ABSOLUTE_QUICK, &&TARGET_LOAD_ATTR_ADAPTIVE, - &&TARGET_LOAD_ATTR_SPLIT_KEYS, + &&TARGET_LOAD_ATTR_INSTANCE_VALUE, &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LOAD_ATTR_SLOT, &&TARGET_LOAD_ATTR_MODULE, @@ -87,7 +87,7 @@ static void *opcode_targets[256] = { &&TARGET_SETUP_ANNOTATIONS, &&TARGET_YIELD_VALUE, &&TARGET_LOAD_METHOD_MODULE, - &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_POP_EXCEPT, &&TARGET_STORE_NAME, &&TARGET_DELETE_NAME, @@ -119,36 +119,36 @@ static void *opcode_targets[256] = { &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, - &&TARGET_STORE_ATTR_SPLIT_KEYS, + &&TARGET_STORE_ATTR_ADAPTIVE, &&TARGET_JUMP_IF_NOT_EXC_MATCH, + &&TARGET_STORE_ATTR_INSTANCE_VALUE, &&TARGET_STORE_ATTR_SLOT, - &&TARGET_STORE_ATTR_WITH_HINT, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, + &&TARGET_STORE_ATTR_WITH_HINT, &&TARGET_LOAD_FAST__LOAD_FAST, - &&TARGET_STORE_FAST__LOAD_FAST, &&TARGET_GEN_START, &&TARGET_RAISE_VARARGS, &&TARGET_CALL_FUNCTION, &&TARGET_MAKE_FUNCTION, &&TARGET_BUILD_SLICE, - &&TARGET_LOAD_FAST__LOAD_CONST, + &&TARGET_STORE_FAST__LOAD_FAST, &&TARGET_MAKE_CELL, &&TARGET_LOAD_CLOSURE, &&TARGET_LOAD_DEREF, &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, - &&TARGET_LOAD_CONST__LOAD_FAST, + &&TARGET_LOAD_FAST__LOAD_CONST, &&TARGET_CALL_FUNCTION_KW, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_STORE_FAST__STORE_FAST, + &&TARGET_LOAD_CONST__LOAD_FAST, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, - &&_unknown_opcode, + &&TARGET_STORE_FAST__STORE_FAST, &&_unknown_opcode, &&_unknown_opcode, &&TARGET_MATCH_CLASS, diff --git a/Python/specialize.c b/Python/specialize.c index 4e025384a6252..6efee7643a455 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -4,6 +4,7 @@ #include "pycore_dict.h" #include "pycore_long.h" #include "pycore_moduleobject.h" +#include "pycore_object.h" #include "opcode.h" #include "structmember.h" // struct PyMemberDef, T_OFFSET_EX @@ -462,6 +463,7 @@ specialize_module_load_attr( PyObject *value = NULL; PyObject *getattr; _Py_IDENTIFIER(__getattr__); + assert(owner->ob_type->tp_inline_values_offset == 0); PyDictObject *dict = (PyDictObject *)m->md_dict; if (dict == NULL) { SPECIALIZATION_FAIL(opcode, SPEC_FAIL_NO_DICT); @@ -584,7 +586,7 @@ specialize_dict_access( PyObject *owner, _Py_CODEUNIT *instr, PyTypeObject *type, DesciptorClassification kind, PyObject *name, _PyAdaptiveEntry *cache0, _PyAttrCache *cache1, - int base_op, int split_op, int hint_op) + int base_op, int values_op, int hint_op) { assert(kind == NON_OVERRIDING || kind == NON_DESCRIPTOR || kind == ABSENT || kind == BUILTIN_CLASSMETHOD || kind == PYTHON_CLASSMETHOD); @@ -595,17 +597,11 @@ specialize_dict_access( } if (type->tp_dictoffset > 0) { PyObject **dictptr = (PyObject **) ((char *)owner + type->tp_dictoffset); - if (*dictptr == NULL || !PyDict_CheckExact(*dictptr)) { - SPECIALIZATION_FAIL(base_op, SPEC_FAIL_NO_DICT); - return 0; - } - // We found an instance with a __dict__. PyDictObject *dict = (PyDictObject *)*dictptr; - PyDictKeysObject *keys = dict->ma_keys; - if ((type->tp_flags & Py_TPFLAGS_HEAPTYPE) - && keys == ((PyHeapTypeObject*)type)->ht_cached_keys - ) { - // Keys are shared + if (type->tp_inline_values_offset && dict == NULL) { + // Virtual dictionary + PyDictKeysObject *keys = ((PyHeapTypeObject *)type)->ht_cached_keys; + assert(type->tp_inline_values_offset > 0); assert(PyUnicode_CheckExact(name)); Py_ssize_t index = _PyDictKeys_StringLookup(keys, name); assert (index != DKIX_ERROR); @@ -613,18 +609,17 @@ specialize_dict_access( SPECIALIZATION_FAIL(base_op, SPEC_FAIL_OUT_OF_RANGE); return 0; } - uint32_t keys_version = _PyDictKeys_GetVersionForCurrentState(keys); - if (keys_version == 0) { - SPECIALIZATION_FAIL(base_op, SPEC_FAIL_OUT_OF_VERSIONS); - return 0; - } - cache1->dk_version_or_hint = keys_version; cache1->tp_version = type->tp_version_tag; cache0->index = (uint16_t)index; - *instr = _Py_MAKECODEUNIT(split_op, _Py_OPARG(*instr)); + *instr = _Py_MAKECODEUNIT(values_op, _Py_OPARG(*instr)); return 0; } else { + if (dict == NULL || !PyDict_CheckExact(dict)) { + SPECIALIZATION_FAIL(base_op, SPEC_FAIL_NO_DICT); + return 0; + } + // We found an instance with a __dict__. PyObject *value = NULL; Py_ssize_t hint = _PyDict_GetItemHint(dict, name, -1, &value); @@ -736,7 +731,7 @@ _Py_Specialize_LoadAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, Sp } int err = specialize_dict_access( owner, instr, type, kind, name, cache0, cache1, - LOAD_ATTR, LOAD_ATTR_SPLIT_KEYS, LOAD_ATTR_WITH_HINT + LOAD_ATTR, LOAD_ATTR_INSTANCE_VALUE, LOAD_ATTR_WITH_HINT ); if (err < 0) { return -1; @@ -818,7 +813,7 @@ _Py_Specialize_StoreAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, S int err = specialize_dict_access( owner, instr, type, kind, name, cache0, cache1, - STORE_ATTR, STORE_ATTR_SPLIT_KEYS, STORE_ATTR_WITH_HINT + STORE_ATTR, STORE_ATTR_INSTANCE_VALUE, STORE_ATTR_WITH_HINT ); if (err < 0) { return -1; @@ -875,6 +870,27 @@ load_method_fail_kind(DesciptorClassification kind) } #endif +static int +specialize_class_load_method(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, + _PyAttrCache *cache1, _PyObjectCache *cache2) +{ + + PyObject *descr = NULL; + DesciptorClassification kind = 0; + kind = analyze_descriptor((PyTypeObject *)owner, name, &descr, 0); + switch (kind) { + case METHOD: + case NON_DESCRIPTOR: + cache1->tp_version = ((PyTypeObject *)owner)->tp_version_tag; + cache2->obj = descr; + *instr = _Py_MAKECODEUNIT(LOAD_METHOD_CLASS, _Py_OPARG(*instr)); + return 0; + default: + SPECIALIZATION_FAIL(LOAD_METHOD, load_method_fail_kind(kind)); + return -1; + } +} + // Please collect stats carefully before and after modifying. A subtle change // can cause a significant drop in cache hits. A possible test is // python.exe -m test_typing test_re test_dis test_zlib. @@ -886,7 +902,6 @@ _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, _PyObjectCache *cache2 = &cache[-2].obj; PyTypeObject *owner_cls = Py_TYPE(owner); - PyDictObject *owner_dict = NULL; if (PyModule_CheckExact(owner)) { int err = specialize_module_load_attr(owner, instr, name, cache0, cache1, LOAD_METHOD, LOAD_METHOD_MODULE); @@ -900,9 +915,12 @@ _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, return -1; } } - if (Py_TYPE(owner_cls)->tp_dictoffset < 0) { - SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_OUT_OF_RANGE); - goto fail; + if (PyType_Check(owner)) { + int err = specialize_class_load_method(owner, instr, name, cache1, cache2); + if (err) { + goto fail; + } + goto success; } // Technically this is fine for bound method calls, but it's uncommon and // slightly slower at runtime to get dict. @@ -910,66 +928,45 @@ _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_OUT_OF_RANGE); goto fail; } - PyObject **owner_dictptr = _PyObject_GetDictPtr(owner); - int owner_has_dict = (owner_dictptr != NULL && *owner_dictptr != NULL); - owner_dict = owner_has_dict ? (PyDictObject *)*owner_dictptr : NULL; - // Make sure dict doesn't get GC-ed halfway. - Py_XINCREF(owner_dict); - // Check for classmethods. - int owner_is_class = PyType_Check(owner); - owner_cls = owner_is_class ? (PyTypeObject *)owner : owner_cls; - - if ((owner_cls->tp_flags & Py_TPFLAGS_VALID_VERSION_TAG) == 0 || - owner_cls->tp_version_tag == 0) { - SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_OUT_OF_VERSIONS); - goto fail; - } PyObject *descr = NULL; DesciptorClassification kind = 0; kind = analyze_descriptor(owner_cls, name, &descr, 0); - // Store the version right away, in case it's modified halfway through. - cache1->tp_version = owner_cls->tp_version_tag; - assert(descr != NULL || kind == ABSENT || kind == GETSET_OVERRIDDEN); if (kind != METHOD) { SPECIALIZATION_FAIL(LOAD_METHOD, load_method_fail_kind(kind)); goto fail; } - // If o.__dict__ changes, the method might be found in o.__dict__ - // instead of old type lookup. So record o.__dict__'s keys. - uint32_t keys_version = UINT32_MAX; - if (owner_has_dict) { - // _PyDictKeys_GetVersionForCurrentState isn't accurate for - // custom dict subclasses at the moment. - if (!PyDict_CheckExact(owner_dict)) { - SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_DICT_SUBCLASS); + if (owner_cls->tp_inline_values_offset) { + PyObject **owner_dictptr = _PyObject_DictPointer(owner); + assert(owner_dictptr); + if (*owner_dictptr) { + SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_IS_ATTR); goto fail; } - assert(PyUnicode_CheckExact(name)); - Py_hash_t hash = PyObject_Hash(name); - if (hash == -1) { - return -1; - } - PyObject *value = NULL; - if (!owner_is_class) { - // Instance methods shouldn't be in o.__dict__. That makes - // it an attribute. - Py_ssize_t ix = _Py_dict_lookup(owner_dict, name, hash, &value); - assert(ix != DKIX_ERROR); - if (ix != DKIX_EMPTY) { - SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_IS_ATTR); - goto fail; - } + PyDictKeysObject *keys = ((PyHeapTypeObject *)owner_cls)->ht_cached_keys; + Py_ssize_t index = _PyDictKeys_StringLookup(keys, name); + if (index != DKIX_EMPTY) { + SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_IS_ATTR); + goto fail; } - keys_version = _PyDictKeys_GetVersionForCurrentState(owner_dict->ma_keys); + uint32_t keys_version = _PyDictKeys_GetVersionForCurrentState(keys); if (keys_version == 0) { - SPECIALIZATION_FAIL(LOAD_ATTR, SPEC_FAIL_OUT_OF_VERSIONS); + SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_OUT_OF_VERSIONS); goto fail; } - // Fall through. - } // Else owner is maybe a builtin with no dict, or __slots__. Doesn't matter. - + cache1->dk_version_or_hint = keys_version; + *instr = _Py_MAKECODEUNIT(LOAD_METHOD_CACHED, _Py_OPARG(*instr)); + } + else { + if (owner_cls->tp_dictoffset == 0) { + *instr = _Py_MAKECODEUNIT(LOAD_METHOD_NO_DICT, _Py_OPARG(*instr)); + } + else { + SPECIALIZATION_FAIL(LOAD_METHOD, SPEC_FAIL_IS_ATTR); + goto fail; + } + } /* `descr` is borrowed. This is safe for methods (even inherited ones from * super classes!) as long as tp_version_tag is validated for two main reasons: * @@ -984,19 +981,15 @@ _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, * PyType_Modified usages in typeobject.c). The MCACHE has been * working since Python 2.6 and it's battle-tested. */ + cache1->tp_version = owner_cls->tp_version_tag; cache2->obj = descr; - cache1->dk_version_or_hint = keys_version; - *instr = _Py_MAKECODEUNIT(owner_is_class ? LOAD_METHOD_CLASS : - LOAD_METHOD_CACHED, _Py_OPARG(*instr)); // Fall through. success: - Py_XDECREF(owner_dict); STAT_INC(LOAD_METHOD, specialization_success); assert(!PyErr_Occurred()); cache0->counter = saturating_start(); return 0; fail: - Py_XDECREF(owner_dict); STAT_INC(LOAD_METHOD, specialization_failure); assert(!PyErr_Occurred()); cache_backoff(cache0); diff --git a/Tools/gdb/libpython.py b/Tools/gdb/libpython.py index a118d326dc00e..54f72320cdd8b 100755 --- a/Tools/gdb/libpython.py +++ b/Tools/gdb/libpython.py @@ -445,10 +445,11 @@ def _write_instance_repr(out, visited, name, pyop_attrdict, address): out.write(name) # Write dictionary of instance attributes: - if isinstance(pyop_attrdict, PyDictObjectPtr): + if isinstance(pyop_attrdict, (PyKeysValuesPair, PyDictObjectPtr)): out.write('(') first = True - for pyop_arg, pyop_val in pyop_attrdict.iteritems(): + items = pyop_attrdict.iteritems() + for pyop_arg, pyop_val in items: if not first: out.write(', ') first = False @@ -520,6 +521,25 @@ def get_attr_dict(self): # Not found, or some kind of error: return None + def get_keys_values(self): + typeobj = self.type() + values_offset = int_from_int(typeobj.field('tp_inline_values_offset')) + if values_offset == 0: + return None + charptr = self._gdbval.cast(_type_char_ptr()) + values_offset + PyDictValuesPtrPtr = gdb.lookup_type("PyDictValues").pointer().pointer() + valuesptr = charptr.cast(PyDictValuesPtrPtr) + values = valuesptr.dereference() + if long(values) == 0: + return None + values = values['values'] + return PyKeysValuesPair(self.get_cached_keys(), values) + + def get_cached_keys(self): + typeobj = self.type() + HeapTypePtr = gdb.lookup_type("PyHeapTypeObject").pointer() + return typeobj._gdbval.cast(HeapTypePtr)['ht_cached_keys'] + def proxyval(self, visited): ''' Support for classes. @@ -533,7 +553,10 @@ def proxyval(self, visited): visited.add(self.as_address()) pyop_attr_dict = self.get_attr_dict() - if pyop_attr_dict: + keys_values = self.get_keys_values() + if keys_values: + attr_dict = keys_values.proxyval(visited) + elif pyop_attr_dict: attr_dict = pyop_attr_dict.proxyval(visited) else: attr_dict = {} @@ -549,9 +572,11 @@ def write_repr(self, out, visited): return visited.add(self.as_address()) - pyop_attrdict = self.get_attr_dict() + pyop_attrs = self.get_keys_values() + if not pyop_attrs: + pyop_attrs = self.get_attr_dict() _write_instance_repr(out, visited, - self.safe_tp_name(), pyop_attrdict, self.as_address()) + self.safe_tp_name(), pyop_attrs, self.as_address()) class ProxyException(Exception): def __init__(self, tp_name, args): @@ -673,6 +698,32 @@ def addr2line(self, addrq): assert False, "Unreachable" +def items_from_keys_and_values(keys, values): + entries, nentries = PyDictObjectPtr._get_entries(keys) + for i in safe_range(nentries): + ep = entries[i] + pyop_value = PyObjectPtr.from_pyobject_ptr(values[i]) + if not pyop_value.is_null(): + pyop_key = PyObjectPtr.from_pyobject_ptr(ep['me_key']) + yield (pyop_key, pyop_value) + +class PyKeysValuesPair: + + def __init__(self, keys, values): + self.keys = keys + self.values = values + + def iteritems(self): + return items_from_keys_and_values(self.keys, self.values) + + def proxyval(self, visited): + result = {} + for pyop_key, pyop_value in self.iteritems(): + proxy_key = pyop_key.proxyval(visited) + proxy_value = pyop_value.proxyval(visited) + result[proxy_key] = proxy_value + return result + class PyDictObjectPtr(PyObjectPtr): """ Class wrapping a gdb.Value that's a PyDictObject* i.e. a dict instance @@ -690,13 +741,14 @@ def iteritems(self): has_values = long(values) if has_values: values = values['values'] + if has_values: + for item in items_from_keys_and_values(keys, values): + yield item + return entries, nentries = self._get_entries(keys) for i in safe_range(nentries): ep = entries[i] - if has_values: - pyop_value = PyObjectPtr.from_pyobject_ptr(values[i]) - else: - pyop_value = PyObjectPtr.from_pyobject_ptr(ep['me_value']) + pyop_value = PyObjectPtr.from_pyobject_ptr(ep['me_value']) if not pyop_value.is_null(): pyop_key = PyObjectPtr.from_pyobject_ptr(ep['me_key']) yield (pyop_key, pyop_value) @@ -732,7 +784,8 @@ def write_repr(self, out, visited): pyop_value.write_repr(out, visited) out.write('}') - def _get_entries(self, keys): + @staticmethod + def _get_entries(keys): dk_nentries = int(keys['dk_nentries']) dk_size = 1< https://github.com/python/cpython/commit/bbe7497c5a44c2b4ec726605cf5a9086ba02daf1 commit: bbe7497c5a44c2b4ec726605cf5a9086ba02daf1 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T15:22:35+02:00 summary: bpo-45434: Remove pystrhex.h header file (GH-28923) Move Include/pystrhex.h to Include/internal/pycore_strhex.h. The header file only contains private functions. The following C extensions are now built with Py_BUILD_CORE_MODULE macro defined to get access to the internal C API: * _blake2 * _hashopenssl * _md5 * _sha1 * _sha3 * _ssl * binascii files: A Include/internal/pycore_strhex.h A Misc/NEWS.d/next/C API/2021-10-13-14-42-46.bpo-45434.INNEEt.rst D Include/pystrhex.h M Doc/whatsnew/3.11.rst M Makefile.pre.in M Modules/Setup M Modules/_blake2/blake2b_impl.c M Modules/_blake2/blake2s_impl.c M Modules/_hashopenssl.c M Modules/_sha3/sha3module.c M Modules/binascii.c M Modules/md5module.c M Modules/sha1module.c M Modules/sha256module.c M Modules/sha512module.c M Objects/bytearrayobject.c M Objects/bytesobject.c M Objects/memoryobject.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/pystrhex.c M setup.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 21a46b4b72133..21a0e1ab9c693 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -598,3 +598,7 @@ Removed since Python 3.3. Use ``PyUnicode_CopyCharacters()`` or ``memcpy()`` (``wchar_t*`` string), and ``PyUnicode_Fill()`` functions instead. (Contributed by Victor Stinner in :issue:`41123`.) + +* Remove the ``pystrhex.h`` header file. It only contains private functions. + C extensions should only include the main ```` header file. + (Contributed by Victor Stinner in :issue:`45434`.) diff --git a/Include/internal/pycore_strhex.h b/Include/internal/pycore_strhex.h new file mode 100644 index 0000000000000..1633671da0f4c --- /dev/null +++ b/Include/internal/pycore_strhex.h @@ -0,0 +1,36 @@ +#ifndef Py_INTERNAL_STRHEX_H +#define Py_INTERNAL_STRHEX_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +// Returns a str() containing the hex representation of argbuf. +PyAPI_FUNC(PyObject*) _Py_strhex(const + char* argbuf, + const Py_ssize_t arglen); + +// Returns a bytes() containing the ASCII hex representation of argbuf. +PyAPI_FUNC(PyObject*) _Py_strhex_bytes( + const char* argbuf, + const Py_ssize_t arglen); + +// These variants include support for a separator between every N bytes: +PyAPI_FUNC(PyObject*) _Py_strhex_with_sep( + const char* argbuf, + const Py_ssize_t arglen, + const PyObject* sep, + const int bytes_per_group); +PyAPI_FUNC(PyObject*) _Py_strhex_bytes_with_sep( + const char* argbuf, + const Py_ssize_t arglen, + const PyObject* sep, + const int bytes_per_group); + +#ifdef __cplusplus +} +#endif +#endif /* !Py_INTERNAL_STRHEX_H */ diff --git a/Include/pystrhex.h b/Include/pystrhex.h deleted file mode 100644 index a4f36305bac69..0000000000000 --- a/Include/pystrhex.h +++ /dev/null @@ -1,22 +0,0 @@ -#ifndef Py_STRHEX_H -#define Py_STRHEX_H - -#ifdef __cplusplus -extern "C" { -#endif - -#ifndef Py_LIMITED_API -/* Returns a str() containing the hex representation of argbuf. */ -PyAPI_FUNC(PyObject*) _Py_strhex(const char* argbuf, const Py_ssize_t arglen); -/* Returns a bytes() containing the ASCII hex representation of argbuf. */ -PyAPI_FUNC(PyObject*) _Py_strhex_bytes(const char* argbuf, const Py_ssize_t arglen); -/* These variants include support for a separator between every N bytes: */ -PyAPI_FUNC(PyObject*) _Py_strhex_with_sep(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, const int bytes_per_group); -PyAPI_FUNC(PyObject*) _Py_strhex_bytes_with_sep(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, const int bytes_per_group); -#endif /* !Py_LIMITED_API */ - -#ifdef __cplusplus -} -#endif - -#endif /* !Py_STRHEX_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index 4d4076f388260..041bc63e2fb8e 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1179,7 +1179,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/pyport.h \ $(srcdir)/Include/pystate.h \ $(srcdir)/Include/pystrcmp.h \ - $(srcdir)/Include/pystrhex.h \ $(srcdir)/Include/pystrtod.h \ $(srcdir)/Include/pythonrun.h \ $(srcdir)/Include/pythread.h \ @@ -1271,6 +1270,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_pymem.h \ $(srcdir)/Include/internal/pycore_pystate.h \ $(srcdir)/Include/internal/pycore_runtime.h \ + $(srcdir)/Include/internal/pycore_strhex.h \ $(srcdir)/Include/internal/pycore_structseq.h \ $(srcdir)/Include/internal/pycore_symtable.h \ $(srcdir)/Include/internal/pycore_sysmodule.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-13-14-42-46.bpo-45434.INNEEt.rst b/Misc/NEWS.d/next/C API/2021-10-13-14-42-46.bpo-45434.INNEEt.rst new file mode 100644 index 0000000000000..933f80670f1ae --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-13-14-42-46.bpo-45434.INNEEt.rst @@ -0,0 +1,3 @@ +Remove the ``pystrhex.h`` header file. It only contains private functions. C +extensions should only include the main ```` header file. Patch by +Victor Stinner. diff --git a/Modules/Setup b/Modules/Setup index 5e26472446677..ba8c154ee4586 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -212,19 +212,23 @@ _symtable symtablemodule.c # OPENSSL=/path/to/openssl/directory # _ssl _ssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -lssl -lcrypto +# -lssl -lcrypto \ +# -DPy_BUILD_CORE_BUILTIN #_hashlib _hashopenssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -lcrypto +# -lcrypto \ +# -DPy_BUILD_CORE_BUILTIN # To statically link OpenSSL: # _ssl _ssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ # -l:libssl.a -Wl,--exclude-libs,libssl.a \ -# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a +# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ +# -DPy_BUILD_CORE_BUILTIN #_hashlib _hashopenssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a +# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ +# -DPy_BUILD_CORE_BUILTIN # The crypt module is now disabled by default because it breaks builds # on many systems (where -lcrypt is needed), e.g. Linux (I believe). @@ -255,18 +259,18 @@ _symtable symtablemodule.c # The _md5 module implements the RSA Data Security, Inc. MD5 # Message-Digest Algorithm, described in RFC 1321. -#_md5 md5module.c +#_md5 md5module.c -DPy_BUILD_CORE_BUILTIN # The _sha module implements the SHA checksum algorithms. # (NIST's Secure Hash Algorithms.) -#_sha1 sha1module.c +#_sha1 sha1module.c -DPy_BUILD_CORE_BUILTIN #_sha256 sha256module.c -DPy_BUILD_CORE_BUILTIN #_sha512 sha512module.c -DPy_BUILD_CORE_BUILTIN -#_sha3 _sha3/sha3module.c +#_sha3 _sha3/sha3module.c -DPy_BUILD_CORE_BUILTIN # _blake module -#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c +#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c -DPy_BUILD_CORE_BUILTIN # The _tkinter module. # @@ -339,7 +343,7 @@ _symtable symtablemodule.c # Helper module for various ascii-encoders -#binascii binascii.c +#binascii binascii.c -DPy_BUILD_CORE_MODULE # Andrew Kuchling's zlib module. # This require zlib 1.1.3 (or later). diff --git a/Modules/_blake2/blake2b_impl.c b/Modules/_blake2/blake2b_impl.c index 8b0d60d02ac7c..b16324e8f7aeb 100644 --- a/Modules/_blake2/blake2b_impl.c +++ b/Modules/_blake2/blake2b_impl.c @@ -14,7 +14,7 @@ */ #include "Python.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() #include "../hashlib.h" #include "blake2ns.h" diff --git a/Modules/_blake2/blake2s_impl.c b/Modules/_blake2/blake2s_impl.c index 7ab3917aa1a65..6b31a363ea27b 100644 --- a/Modules/_blake2/blake2s_impl.c +++ b/Modules/_blake2/blake2s_impl.c @@ -14,7 +14,7 @@ */ #include "Python.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() #include "../hashlib.h" #include "blake2ns.h" diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c index b9e68c05c3edb..b4ba60baaac93 100644 --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -22,7 +22,7 @@ #include "Python.h" #include "hashlib.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() /* EVP is the preferred interface to hashing in OpenSSL */ #include diff --git a/Modules/_sha3/sha3module.c b/Modules/_sha3/sha3module.c index 3974e0b6b47fa..a033c4e452545 100644 --- a/Modules/_sha3/sha3module.c +++ b/Modules/_sha3/sha3module.c @@ -16,7 +16,7 @@ */ #include "Python.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() #include "../hashlib.h" /* ************************************************************************** diff --git a/Modules/binascii.c b/Modules/binascii.c index db960dcdcb0b6..ef209881494ac 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -56,9 +56,9 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex_bytes_with_sep() #ifdef USE_ZLIB_CRC32 -#include "zlib.h" +# include "zlib.h" #endif typedef struct binascii_state { diff --git a/Modules/md5module.c b/Modules/md5module.c index b50d2e5b0bdb7..4d03f6b844b33 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -18,7 +18,7 @@ #include "Python.h" #include "hashlib.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() /*[clinic input] module _md5 diff --git a/Modules/sha1module.c b/Modules/sha1module.c index d186aa460a07b..153bc12a8bd74 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -18,7 +18,7 @@ #include "Python.h" #include "hashlib.h" -#include "pystrhex.h" +#include "pycore_strhex.h" // _Py_strhex() /*[clinic input] module _sha1 diff --git a/Modules/sha256module.c b/Modules/sha256module.c index fac7073d951d2..5858071db4e13 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -18,9 +18,9 @@ #include "Python.h" #include "pycore_bitutils.h" // _Py_bswap32() +#include "pycore_strhex.h" // _Py_strhex() #include "structmember.h" // PyMemberDef #include "hashlib.h" -#include "pystrhex.h" /*[clinic input] module _sha256 diff --git a/Modules/sha512module.c b/Modules/sha512module.c index 4ea2fc16de052..e50b69be4617d 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -18,9 +18,9 @@ #include "Python.h" #include "pycore_bitutils.h" // _Py_bswap64() +#include "pycore_strhex.h" // _Py_strhex() #include "structmember.h" // PyMemberDef #include "hashlib.h" -#include "pystrhex.h" /*[clinic input] module _sha512 diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index 1ab9621b1f265..a6009854221ff 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -4,9 +4,9 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_bytes_methods.h" -#include "pycore_object.h" +#include "pycore_object.h" // _PyObject_GC_UNTRACK() +#include "pycore_strhex.h" // _Py_strhex_with_sep() #include "bytesobject.h" -#include "pystrhex.h" /*[clinic input] class bytearray "PyByteArrayObject *" "&PyByteArray_Type" diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index bc0b075cf405b..1163cf00034b2 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -10,8 +10,8 @@ #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_object.h" // _PyObject_GC_TRACK #include "pycore_pymem.h" // PYMEM_CLEANBYTE +#include "pycore_strhex.h" // _Py_strhex_with_sep() -#include "pystrhex.h" #include /*[clinic input] diff --git a/Objects/memoryobject.c b/Objects/memoryobject.c index 913d358062219..6257455d34715 100644 --- a/Objects/memoryobject.c +++ b/Objects/memoryobject.c @@ -11,10 +11,10 @@ */ #include "Python.h" -#include "pycore_abstract.h" // _PyIndex_Check() -#include "pycore_object.h" -#include "pystrhex.h" -#include +#include "pycore_abstract.h" // _PyIndex_Check() +#include "pycore_object.h" // _PyObject_GC_UNTRACK() +#include "pycore_strhex.h" // _Py_strhex_with_sep() +#include // offsetof() /*[clinic input] class memoryview "PyMemoryViewObject *" "&PyMemoryView_Type" diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index b8cadf469355f..d0e1b52a4927c 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -209,6 +209,7 @@ + @@ -248,7 +249,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 4eccf4f43d1ad..9370f405132c3 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -201,9 +201,6 @@ Include - - Include - Include @@ -588,6 +585,9 @@ Include\internal + + Include\internal + Include\internal diff --git a/Python/pystrhex.c b/Python/pystrhex.c index b74e57ad913b1..f7fe3b6eb85d0 100644 --- a/Python/pystrhex.c +++ b/Python/pystrhex.c @@ -1,8 +1,8 @@ -/* bytes to hex implementation */ +/* Format bytes as hexadecimal */ #include "Python.h" +#include "pycore_strhex.h" // _Py_strhex_with_sep() -#include "pystrhex.h" static PyObject *_Py_strhex_impl(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, int bytes_per_sep_group, diff --git a/setup.py b/setup.py index 039c96b0e4d7b..56c06cbddaf21 100644 --- a/setup.py +++ b/setup.py @@ -1709,12 +1709,12 @@ def detect_compress_exts(self): # Helper module for various ascii-encoders. Uses zlib for an optimized # crc32 if we have it. Otherwise binascii uses its own. + extra_compile_args = ['-DPy_BUILD_CORE_MODULE'] if have_zlib: - extra_compile_args = ['-DUSE_ZLIB_CRC32'] + extra_compile_args.append('-DUSE_ZLIB_CRC32') libraries = ['z'] extra_link_args = zlib_extra_link_args else: - extra_compile_args = [] libraries = [] extra_link_args = [] self.add(Extension('binascii', ['binascii.c'], @@ -2469,6 +2469,7 @@ def split_var(name, sep): library_dirs=openssl_libdirs, libraries=openssl_libs, runtime_library_dirs=runtime_library_dirs, + extra_compile_args=['-DPy_BUILD_CORE_MODULE'], ) # This static linking is NOT OFFICIALLY SUPPORTED. @@ -2530,27 +2531,29 @@ def detect_hash_builtins(self): if "sha256" in configured: self.add(Extension( '_sha256', ['sha256module.c'], + depends=['hashlib.h'], extra_compile_args=['-DPy_BUILD_CORE_MODULE'], - depends=['hashlib.h'] )) if "sha512" in configured: self.add(Extension( '_sha512', ['sha512module.c'], + depends=['hashlib.h'], extra_compile_args=['-DPy_BUILD_CORE_MODULE'], - depends=['hashlib.h'] )) if "md5" in configured: self.add(Extension( '_md5', ['md5module.c'], - depends=['hashlib.h'] + depends=['hashlib.h'], + extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "sha1" in configured: self.add(Extension( '_sha1', ['sha1module.c'], - depends=['hashlib.h'] + depends=['hashlib.h'], + extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "blake2" in configured: @@ -2565,7 +2568,8 @@ def detect_hash_builtins(self): '_blake2/blake2b_impl.c', '_blake2/blake2s_impl.c' ], - depends=blake2_deps + depends=blake2_deps, + extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "sha3" in configured: @@ -2576,7 +2580,8 @@ def detect_hash_builtins(self): self.add(Extension( '_sha3', ['_sha3/sha3module.c'], - depends=sha3_deps + depends=sha3_deps, + extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) def detect_nis(self): From webhook-mailer at python.org Wed Oct 13 09:38:50 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 13 Oct 2021 13:38:50 -0000 Subject: [Python-checkins] bpo-45256: Fix cleanup of stolen locals for Python-to-Python calls (GH-28905) Message-ID: https://github.com/python/cpython/commit/3901c081143ef29624f9c1cb49cc70a70321d139 commit: 3901c081143ef29624f9c1cb49cc70a70321d139 branch: main author: Pablo Galindo Salgado committer: markshannon date: 2021-10-13T14:38:41+01:00 summary: bpo-45256: Fix cleanup of stolen locals for Python-to-Python calls (GH-28905) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 5cf2ab341ddc4..0af233c0ba485 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5619,6 +5619,30 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, return 0; fail: /* Jump here from prelude on failure */ + if (steal_args) { + // If we failed to initialize locals, make sure the caller still own all the + // arguments that were on the stack. We need to increment the reference count + // of everything we copied (everything in localsplus) that came from the stack + // (everything that is present in the "args" array). + Py_ssize_t kwcount = kwnames != NULL ? PyTuple_GET_SIZE(kwnames) : 0; + for (Py_ssize_t k=0; k < total_args; k++) { + PyObject* arg = localsplus[k]; + for (Py_ssize_t j=0; j < argcount + kwcount; j++) { + if (args[j] == arg) { + Py_XINCREF(arg); + break; + } + } + } + // Restore all the **kwargs we placed into the kwargs dictionary + if (kwdict) { + PyObject *key, *value; + Py_ssize_t pos = 0; + while (PyDict_Next(kwdict, &pos, &key, &value)) { + Py_INCREF(value); + } + } + } return -1; } @@ -5683,16 +5707,6 @@ _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, } PyObject **localsarray = _PyFrame_GetLocalsArray(frame); if (initialize_locals(tstate, con, localsarray, args, argcount, kwnames, steal_args)) { - if (steal_args) { - // If we failed to initialize locals, make sure the caller still own all the - // arguments. Notice that we only need to increase the reference count of the - // *valid* arguments (i.e. the ones that fit into the frame). - PyCodeObject *co = (PyCodeObject*)con->fc_code; - const size_t total_args = co->co_argcount + co->co_kwonlyargcount; - for (size_t i = 0; i < Py_MIN(argcount, total_args); i++) { - Py_XINCREF(frame->localsplus[i]); - } - } _PyFrame_Clear(frame, 0); return NULL; } From webhook-mailer at python.org Wed Oct 13 11:22:19 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 15:22:19 -0000 Subject: [Python-checkins] bpo-45434: Mark the PyTokenizer C API as private (GH-28924) Message-ID: https://github.com/python/cpython/commit/713bb19356bce9b8f2b95461834fe1dae505f889 commit: 713bb19356bce9b8f2b95461834fe1dae505f889 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T17:22:14+02:00 summary: bpo-45434: Mark the PyTokenizer C API as private (GH-28924) Rename PyTokenize functions to mark them as private: * PyTokenizer_FindEncodingFilename() => _PyTokenizer_FindEncodingFilename() * PyTokenizer_FromString() => _PyTokenizer_FromString() * PyTokenizer_FromFile() => _PyTokenizer_FromFile() * PyTokenizer_FromUTF8() => _PyTokenizer_FromUTF8() * PyTokenizer_Free() => _PyTokenizer_Free() * PyTokenizer_Get() => _PyTokenizer_Get() Remove the unused PyTokenizer_FindEncoding() function. import.c: remove unused #include "errcode.h". files: M Parser/pegen.c M Parser/string_parser.c M Parser/tokenizer.c M Parser/tokenizer.h M Python/Python-tokenize.c M Python/import.c M Python/traceback.c diff --git a/Parser/pegen.c b/Parser/pegen.c index a9896356e5b58..e4d2692539264 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -729,7 +729,7 @@ _PyPegen_fill_token(Parser *p) { const char *start; const char *end; - int type = PyTokenizer_Get(p->tok, &start, &end); + int type = _PyTokenizer_Get(p->tok, &start, &end); // Record and skip '# type: ignore' comments while (type == TYPE_IGNORE) { @@ -746,7 +746,7 @@ _PyPegen_fill_token(Parser *p) PyErr_NoMemory(); return -1; } - type = PyTokenizer_Get(p->tok, &start, &end); + type = _PyTokenizer_Get(p->tok, &start, &end); } // If we have reached the end and we are in single input mode we need to insert a newline and reset the parsing @@ -1306,7 +1306,7 @@ _PyPegen_check_tokenizer_errors(Parser *p) { for (;;) { const char *start; const char *end; - switch (PyTokenizer_Get(p->tok, &start, &end)) { + switch (_PyTokenizer_Get(p->tok, &start, &end)) { case ERRORTOKEN: if (p->tok->level != 0) { int error_lineno = p->tok->parenlinenostack[p->tok->level-1]; @@ -1411,7 +1411,7 @@ _PyPegen_run_parser_from_file_pointer(FILE *fp, int start_rule, PyObject *filena const char *enc, const char *ps1, const char *ps2, PyCompilerFlags *flags, int *errcode, PyArena *arena) { - struct tok_state *tok = PyTokenizer_FromFile(fp, enc, ps1, ps2); + struct tok_state *tok = _PyTokenizer_FromFile(fp, enc, ps1, ps2); if (tok == NULL) { if (PyErr_Occurred()) { raise_tokenizer_init_error(filename_ob); @@ -1441,7 +1441,7 @@ _PyPegen_run_parser_from_file_pointer(FILE *fp, int start_rule, PyObject *filena _PyPegen_Parser_Free(p); error: - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return result; } @@ -1453,9 +1453,9 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen struct tok_state *tok; if (flags == NULL || flags->cf_flags & PyCF_IGNORE_COOKIE) { - tok = PyTokenizer_FromUTF8(str, exec_input); + tok = _PyTokenizer_FromUTF8(str, exec_input); } else { - tok = PyTokenizer_FromString(str, exec_input); + tok = _PyTokenizer_FromString(str, exec_input); } if (tok == NULL) { if (PyErr_Occurred()) { @@ -1483,7 +1483,7 @@ _PyPegen_run_parser_from_string(const char *str, int start_rule, PyObject *filen _PyPegen_Parser_Free(p); error: - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return result; } diff --git a/Parser/string_parser.c b/Parser/string_parser.c index 2880d07e5f6fe..cffe24e992e44 100644 --- a/Parser/string_parser.c +++ b/Parser/string_parser.c @@ -386,7 +386,7 @@ fstring_compile_expr(Parser *p, const char *expr_start, const char *expr_end, str[0] = '('; str[len+1] = ')'; - struct tok_state* tok = PyTokenizer_FromString(str, 1); + struct tok_state* tok = _PyTokenizer_FromString(str, 1); if (tok == NULL) { PyMem_Free(str); return NULL; @@ -409,7 +409,7 @@ fstring_compile_expr(Parser *p, const char *expr_start, const char *expr_end, exit: PyMem_Free(str); _PyPegen_Parser_Free(p2); - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return result; } diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index c7a014de42d8f..ae3874b09de48 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -108,7 +108,7 @@ static char * error_ret(struct tok_state *tok) /* XXX */ { tok->decoding_erred = 1; - if (tok->fp != NULL && tok->buf != NULL) /* see PyTokenizer_Free */ + if (tok->fp != NULL && tok->buf != NULL) /* see _PyTokenizer_Free */ PyMem_Free(tok->buf); tok->buf = tok->cur = tok->inp = NULL; tok->start = NULL; @@ -702,7 +702,7 @@ decode_str(const char *input, int single, struct tok_state *tok) /* Set up tokenizer for string */ struct tok_state * -PyTokenizer_FromString(const char *str, int exec_input) +_PyTokenizer_FromString(const char *str, int exec_input) { struct tok_state *tok = tok_new(); char *decoded; @@ -711,7 +711,7 @@ PyTokenizer_FromString(const char *str, int exec_input) return NULL; decoded = decode_str(str, exec_input, tok); if (decoded == NULL) { - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return NULL; } @@ -723,7 +723,7 @@ PyTokenizer_FromString(const char *str, int exec_input) /* Set up tokenizer for UTF-8 string */ struct tok_state * -PyTokenizer_FromUTF8(const char *str, int exec_input) +_PyTokenizer_FromUTF8(const char *str, int exec_input) { struct tok_state *tok = tok_new(); char *translated; @@ -731,7 +731,7 @@ PyTokenizer_FromUTF8(const char *str, int exec_input) return NULL; tok->input = translated = translate_newlines(str, exec_input, tok); if (translated == NULL) { - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return NULL; } tok->decoding_state = STATE_NORMAL; @@ -739,7 +739,7 @@ PyTokenizer_FromUTF8(const char *str, int exec_input) tok->str = translated; tok->encoding = new_string("utf-8", 5, tok); if (!tok->encoding) { - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return NULL; } @@ -751,14 +751,14 @@ PyTokenizer_FromUTF8(const char *str, int exec_input) /* Set up tokenizer for file */ struct tok_state * -PyTokenizer_FromFile(FILE *fp, const char* enc, - const char *ps1, const char *ps2) +_PyTokenizer_FromFile(FILE *fp, const char* enc, + const char *ps1, const char *ps2) { struct tok_state *tok = tok_new(); if (tok == NULL) return NULL; if ((tok->buf = (char *)PyMem_Malloc(BUFSIZ)) == NULL) { - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return NULL; } tok->cur = tok->inp = tok->buf; @@ -771,7 +771,7 @@ PyTokenizer_FromFile(FILE *fp, const char* enc, gets copied into the parse tree. */ tok->encoding = new_string(enc, strlen(enc), tok); if (!tok->encoding) { - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return NULL; } tok->decoding_state = STATE_NORMAL; @@ -782,7 +782,7 @@ PyTokenizer_FromFile(FILE *fp, const char* enc, /* Free a tok_state structure */ void -PyTokenizer_Free(struct tok_state *tok) +_PyTokenizer_Free(struct tok_state *tok) { if (tok->encoding != NULL) { PyMem_Free(tok->encoding); @@ -2049,7 +2049,8 @@ tok_get(struct tok_state *tok, const char **p_start, const char **p_end) } int -PyTokenizer_Get(struct tok_state *tok, const char **p_start, const char **p_end) +_PyTokenizer_Get(struct tok_state *tok, + const char **p_start, const char **p_end) { int result = tok_get(tok, p_start, p_end); if (tok->decoding_erred) { @@ -2062,7 +2063,7 @@ PyTokenizer_Get(struct tok_state *tok, const char **p_start, const char **p_end) /* Get the encoding of a Python file. Check for the coding cookie and check if the file starts with a BOM. - PyTokenizer_FindEncodingFilename() returns NULL when it can't find the + _PyTokenizer_FindEncodingFilename() returns NULL when it can't find the encoding in the first or second line of the file (in which case the encoding should be assumed to be UTF-8). @@ -2070,7 +2071,7 @@ PyTokenizer_Get(struct tok_state *tok, const char **p_start, const char **p_end) by the caller. */ char * -PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) +_PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) { struct tok_state *tok; FILE *fp; @@ -2087,7 +2088,7 @@ PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) if (fp == NULL) { return NULL; } - tok = PyTokenizer_FromFile(fp, NULL, NULL, NULL); + tok = _PyTokenizer_FromFile(fp, NULL, NULL, NULL); if (tok == NULL) { fclose(fp); return NULL; @@ -2100,12 +2101,12 @@ PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) tok->filename = PyUnicode_FromString(""); if (tok->filename == NULL) { fclose(fp); - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return encoding; } } while (tok->lineno < 2 && tok->done == E_OK) { - PyTokenizer_Get(tok, &p_start, &p_end); + _PyTokenizer_Get(tok, &p_start, &p_end); } fclose(fp); if (tok->encoding) { @@ -2114,18 +2115,11 @@ PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) strcpy(encoding, tok->encoding); } } - PyTokenizer_Free(tok); + _PyTokenizer_Free(tok); return encoding; } -char * -PyTokenizer_FindEncoding(int fd) -{ - return PyTokenizer_FindEncodingFilename(fd, NULL); -} - #ifdef Py_DEBUG - void tok_dump(int type, char *start, char *end) { @@ -2133,5 +2127,4 @@ tok_dump(int type, char *start, char *end) if (type == NAME || type == NUMBER || type == STRING || type == OP) printf("(%.*s)", (int)(end - start), start); } - -#endif +#endif // Py_DEBUG diff --git a/Parser/tokenizer.h b/Parser/tokenizer.h index 677f9dba490be..1d1cfd639d9d5 100644 --- a/Parser/tokenizer.h +++ b/Parser/tokenizer.h @@ -86,12 +86,12 @@ struct tok_state { enum interactive_underflow_t interactive_underflow; }; -extern struct tok_state *PyTokenizer_FromString(const char *, int); -extern struct tok_state *PyTokenizer_FromUTF8(const char *, int); -extern struct tok_state *PyTokenizer_FromFile(FILE *, const char*, +extern struct tok_state *_PyTokenizer_FromString(const char *, int); +extern struct tok_state *_PyTokenizer_FromUTF8(const char *, int); +extern struct tok_state *_PyTokenizer_FromFile(FILE *, const char*, const char *, const char *); -extern void PyTokenizer_Free(struct tok_state *); -extern int PyTokenizer_Get(struct tok_state *, const char **, const char **); +extern void _PyTokenizer_Free(struct tok_state *); +extern int _PyTokenizer_Get(struct tok_state *, const char **, const char **); #define tok_dump _Py_tok_dump diff --git a/Python/Python-tokenize.c b/Python/Python-tokenize.c index fa713282558b0..d3ebbe1331a15 100644 --- a/Python/Python-tokenize.c +++ b/Python/Python-tokenize.c @@ -47,7 +47,7 @@ tokenizeriter_new_impl(PyTypeObject *type, const char *source) if (filename == NULL) { return NULL; } - self->tok = PyTokenizer_FromUTF8(source, 1); + self->tok = _PyTokenizer_FromUTF8(source, 1); if (self->tok == NULL) { Py_DECREF(filename); return NULL; @@ -61,7 +61,7 @@ tokenizeriter_next(tokenizeriterobject *it) { const char *start; const char *end; - int type = PyTokenizer_Get(it->tok, &start, &end); + int type = _PyTokenizer_Get(it->tok, &start, &end); if (type == ERRORTOKEN && PyErr_Occurred()) { return NULL; } @@ -105,7 +105,7 @@ static void tokenizeriter_dealloc(tokenizeriterobject *it) { PyTypeObject *tp = Py_TYPE(it); - PyTokenizer_Free(it->tok); + _PyTokenizer_Free(it->tok); tp->tp_free(it); Py_DECREF(tp); } diff --git a/Python/import.c b/Python/import.c index 731f0f59118d6..4bc1e518bf525 100644 --- a/Python/import.c +++ b/Python/import.c @@ -11,7 +11,6 @@ #include "pycore_interp.h" // _PyInterpreterState_ClearModules() #include "pycore_pystate.h" // _PyInterpreterState_GET() #include "pycore_sysmodule.h" -#include "errcode.h" #include "marshal.h" #include "code.h" #include "importdl.h" diff --git a/Python/traceback.c b/Python/traceback.c index ffa7c3494eafc..b18cbb91ce8ed 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -29,7 +29,7 @@ #define MAX_NTHREADS 100 /* Function from Parser/tokenizer.c */ -extern char * PyTokenizer_FindEncodingFilename(int, PyObject *); +extern char* _PyTokenizer_FindEncodingFilename(int, PyObject *); _Py_IDENTIFIER(TextIOWrapper); _Py_IDENTIFIER(close); @@ -431,7 +431,7 @@ _Py_DisplaySourceLine(PyObject *f, PyObject *filename, int lineno, int indent, i Py_DECREF(binary); return 0; } - found_encoding = PyTokenizer_FindEncodingFilename(fd, filename); + found_encoding = _PyTokenizer_FindEncodingFilename(fd, filename); if (found_encoding == NULL) PyErr_Clear(); encoding = (found_encoding != NULL) ? found_encoding : "utf-8"; From webhook-mailer at python.org Wed Oct 13 11:35:29 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 15:35:29 -0000 Subject: [Python-checkins] bpo-45410: regrtest replaces print_warning.orig_stderr (GH-28926) Message-ID: https://github.com/python/cpython/commit/676201a59f90caace606d11d4172aa74c1cd4992 commit: 676201a59f90caace606d11d4172aa74c1cd4992 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T17:35:21+02:00 summary: bpo-45410: regrtest replaces print_warning.orig_stderr (GH-28926) When running Python tests with -W, runtest() now replaces support.print_warning.orig_stderr to preserve the messages order. Add an unit test. files: M Lib/test/libregrtest/runtest.py M Lib/test/libregrtest/save_env.py M Lib/test/test_regrtest.py diff --git a/Lib/test/libregrtest/runtest.py b/Lib/test/libregrtest/runtest.py index 6fb996a5f947b..23970410a28fb 100644 --- a/Lib/test/libregrtest/runtest.py +++ b/Lib/test/libregrtest/runtest.py @@ -196,10 +196,18 @@ def _runtest(ns: Namespace, test_name: str) -> TestResult: stream = io.StringIO() orig_stdout = sys.stdout orig_stderr = sys.stderr + print_warning = support.print_warning + orig_print_warnings_stderr = print_warning.orig_stderr + output = None try: sys.stdout = stream sys.stderr = stream + # print_warning() writes into the temporary stream to preserve + # messages order. If support.environment_altered becomes true, + # warnings will be written to sys.stderr below. + print_warning.orig_stderr = stream + result = _runtest_inner(ns, test_name, display_failure=False) if not isinstance(result, Passed): @@ -207,6 +215,7 @@ def _runtest(ns: Namespace, test_name: str) -> TestResult: finally: sys.stdout = orig_stdout sys.stderr = orig_stderr + print_warning.orig_stderr = orig_print_warnings_stderr if output is not None: sys.stderr.write(output) diff --git a/Lib/test/libregrtest/save_env.py b/Lib/test/libregrtest/save_env.py index f0bfcf389992d..60c9be24617a6 100644 --- a/Lib/test/libregrtest/save_env.py +++ b/Lib/test/libregrtest/save_env.py @@ -320,7 +320,8 @@ def __exit__(self, exc_type, exc_val, exc_tb): support.environment_altered = True restore(original) if not self.quiet and not self.pgo: - print_warning(f"{name} was modified by {self.testname}") - print(f" Before: {original}\n After: {current} ", - file=sys.stderr, flush=True) + print_warning( + f"{name} was modified by {self.testname}\n" + f" Before: {original}\n" + f" After: {current} ") return False diff --git a/Lib/test/test_regrtest.py b/Lib/test/test_regrtest.py index 3780feeda30e1..dcc795de1a4b6 100644 --- a/Lib/test/test_regrtest.py +++ b/Lib/test/test_regrtest.py @@ -1300,6 +1300,46 @@ def test_threading_excepthook(self): self.assertIn("Warning -- Uncaught thread exception", output) self.assertIn("Exception: bug in thread", output) + def test_print_warning(self): + # bpo-45410: The order of messages must be preserved when -W and + # support.print_warning() are used. + code = textwrap.dedent(r""" + import sys + import unittest + from test import support + + class MyObject: + pass + + def func_bug(): + raise Exception("bug in thread") + + class Tests(unittest.TestCase): + def test_print_warning(self): + print("msg1: stdout") + support.print_warning("msg2: print_warning") + # Fail with ENV CHANGED to see print_warning() log + support.environment_altered = True + """) + testname = self.create_test(code=code) + + # Expect an output like: + # + # test_threading_excepthook (test.test_x.Tests) ... msg1: stdout + # Warning -- msg2: print_warning + # ok + regex = (r"test_print_warning.*msg1: stdout\n" + r"Warning -- msg2: print_warning\n" + r"ok\n") + for option in ("-v", "-W"): + with self.subTest(option=option): + cmd = ["--fail-env-changed", option, testname] + output = self.run_tests(*cmd, exitcode=3) + self.check_executed_tests(output, [testname], + env_changed=[testname], + fail_env_changed=True) + self.assertRegex(output, regex) + def test_unicode_guard_env(self): guard = os.environ.get(setup.UNICODE_GUARD_ENV) self.assertIsNotNone(guard, f"{setup.UNICODE_GUARD_ENV} not set") From webhook-mailer at python.org Wed Oct 13 12:12:53 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:12:53 -0000 Subject: [Python-checkins] bpo-45229: Make tkinter tests discoverable (GH-28637) Message-ID: https://github.com/python/cpython/commit/f59ed3c310a7ceebf2a56a84ea969a7f75d95b64 commit: f59ed3c310a7ceebf2a56a84ea969a7f75d95b64 branch: main author: Serhiy Storchaka committer: ambv date: 2021-10-13T18:12:48+02:00 summary: bpo-45229: Make tkinter tests discoverable (GH-28637) files: D Lib/tkinter/test/runtktests.py D Lib/tkinter/test/test_ttk/test_functions.py M Lib/test/test_tk.py M Lib/test/test_ttk_guionly.py M Lib/test/test_ttk_textonly.py M Lib/tkinter/test/test_tkinter/test_colorchooser.py M Lib/tkinter/test/test_tkinter/test_font.py M Lib/tkinter/test/test_tkinter/test_images.py M Lib/tkinter/test/test_tkinter/test_loadtk.py M Lib/tkinter/test/test_tkinter/test_messagebox.py M Lib/tkinter/test/test_tkinter/test_misc.py M Lib/tkinter/test/test_tkinter/test_simpledialog.py M Lib/tkinter/test/test_tkinter/test_text.py M Lib/tkinter/test/test_tkinter/test_variables.py M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/test/test_ttk/test_style.py M Lib/tkinter/test/test_ttk/test_widgets.py M PCbuild/lib.pyproj diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py index 59842a5e25e36..69cc2322cd9aa 100644 --- a/Lib/test/test_tk.py +++ b/Lib/test/test_tk.py @@ -6,11 +6,9 @@ # Skip test if tk cannot be initialized. support.requires('gui') -from tkinter.test import runtktests +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_tkinter') -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_tkinter'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py index abb26433652f1..8f59839d066e6 100644 --- a/Lib/test/test_ttk_guionly.py +++ b/Lib/test/test_ttk_guionly.py @@ -11,25 +11,26 @@ import tkinter from _tkinter import TclError from tkinter import ttk -from tkinter.test import runtktests - -root = None -try: - root = tkinter.Tk() - button = ttk.Button(root) - button.destroy() - del button -except TclError as msg: - # assuming ttk is not available - raise unittest.SkipTest("ttk not available: %s" % msg) -finally: - if root is not None: - root.destroy() - del root - -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_ttk'])) + + +def setUpModule(): + root = None + try: + root = tkinter.Tk() + button = ttk.Button(root) + button.destroy() + del button + except TclError as msg: + # assuming ttk is not available + raise unittest.SkipTest("ttk not available: %s" % msg) + finally: + if root is not None: + root.destroy() + del root + +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_ttk') + if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_textonly.py b/Lib/test/test_ttk_textonly.py index 013b0e9ee21da..96dc179a69eca 100644 --- a/Lib/test/test_ttk_textonly.py +++ b/Lib/test/test_ttk_textonly.py @@ -1,15 +1,463 @@ -from test import support from test.support import import_helper - # Skip this test if _tkinter does not exist. import_helper.import_module('_tkinter') -from tkinter.test import runtktests +import unittest +from tkinter import ttk + + +class MockTkApp: + + def splitlist(self, arg): + if isinstance(arg, tuple): + return arg + return arg.split(':') + + def wantobjects(self): + return True + + +class MockTclObj(object): + typename = 'test' + + def __init__(self, val): + self.val = val + + def __str__(self): + return str(self.val) + + +class MockStateSpec(object): + typename = 'StateSpec' + + def __init__(self, *args): + self.val = args + + def __str__(self): + return ' '.join(self.val) + + +class InternalFunctionsTest(unittest.TestCase): + + def test_format_optdict(self): + def check_against(fmt_opts, result): + for i in range(0, len(fmt_opts), 2): + self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) + if result: + self.fail("result still got elements: %s" % result) + + # passing an empty dict should return an empty object (tuple here) + self.assertFalse(ttk._format_optdict({})) + + # check list formatting + check_against( + ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), + {'-fg': 'blue', '-padding': '1 2 3 4'}) + + # check tuple formatting (same as list) + check_against( + ttk._format_optdict({'test': (1, 2, '', 0)}), + {'-test': '1 2 {} 0'}) + + # check untouched values + check_against( + ttk._format_optdict({'test': {'left': 'as is'}}), + {'-test': {'left': 'as is'}}) + + # check script formatting + check_against( + ttk._format_optdict( + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) + + opts = {'???': True, '?': False} + orig_opts = opts.copy() + # check if giving unicode keys is fine + check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) + # opts should remain unchanged + self.assertEqual(opts, orig_opts) + + # passing values with spaces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one two', 'three')}), + {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) + + # ignore an option + amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore non-existing options + amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore every option + self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) + + + def test_format_mapdict(self): + opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} + result = ttk._format_mapdict(opts) + self.assertEqual(len(result), len(list(opts.keys())) * 2) + self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) + self.assertEqual(ttk._format_mapdict(opts, script=True), + ('-a', '{{b c} val d otherval {} single}')) + + self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) + + opts = {'?????d?': [('?', 'v?l')]} + result = ttk._format_mapdict(opts) + self.assertEqual(result, ('-?????d?', '? v?l')) + + self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), + ('-opt', '{} value')) + + # empty states + valid = {'opt': [('', '', 'hi')]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) + + # when passing multiple states, they all must be strings + invalid = {'opt': [(1, 2, 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + invalid = {'opt': [([1], '2', 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + # but when passing a single state, it can be anything + valid = {'opt': [[1, 'value']]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) + # special attention to single states which evaluate to False + for stateval in (None, 0, False, '', set()): # just some samples + valid = {'opt': [(stateval, 'value')]} + self.assertEqual(ttk._format_mapdict(valid), + ('-opt', '{} value')) + + # values must be iterable + opts = {'a': None} + self.assertRaises(TypeError, ttk._format_mapdict, opts) + + + def test_format_elemcreate(self): + self.assertTrue(ttk._format_elemcreate(None), (None, ())) + + ## Testing type = image + # image type expects at least an image name, so this should raise + # IndexError since it tries to access the index 0 of an empty tuple + self.assertRaises(IndexError, ttk._format_elemcreate, 'image') + + # don't format returned values as a tcl script + # minimum acceptable for image type + self.assertEqual(ttk._format_elemcreate('image', False, 'test'), + ("test ", ())) + # specifying a state spec + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('', 'a')), ("test {} a", ())) + # state spec with multiple states + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b', 'c')), ("test {a b} c", ())) + # state spec and options + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) + # format returned values as a tcl script + # state spec with multiple states and an option with a multivalue + self.assertEqual(ttk._format_elemcreate('image', True, 'test', + ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) + + ## Testing type = vsapi + # vsapi type expects at least a class name and a part_id, so this + # should raise a ValueError since it tries to get two elements from + # an empty tuple + self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') + + # don't format returned values as a tcl script + # minimum acceptable for vsapi + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), + ("a b ", ())) + # now with a state spec with multiple states + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b', 'c')), ("a b {a b} c", ())) + # state spec and option + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) + # format returned values as a tcl script + # state spec with a multivalue and an option + self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', + ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) + + # Testing type = from + # from type expects at least a type name + self.assertRaises(IndexError, ttk._format_elemcreate, 'from') + + self.assertEqual(ttk._format_elemcreate('from', False, 'a'), + ('a', ())) + self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), + ('a', ('b', ))) + self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), + ('{a}', 'b')) + + + def test_format_layoutlist(self): + def sample(indent=0, indent_size=2): + return ttk._format_layoutlist( + [('a', {'other': [1, 2, 3], 'children': + [('b', {'children': + [('c', {'children': + [('d', {'nice': 'opt'})], 'something': (1, 2) + })] + })] + })], indent=indent, indent_size=indent_size)[0] + + def sample_expected(indent=0, indent_size=2): + spaces = lambda amount=0: ' ' * (amount + indent) + return ( + "%sa -other {1 2 3} -children {\n" + "%sb -children {\n" + "%sc -something {1 2} -children {\n" + "%sd -nice opt\n" + "%s}\n" + "%s}\n" + "%s}" % (spaces(), spaces(indent_size), + spaces(2 * indent_size), spaces(3 * indent_size), + spaces(2 * indent_size), spaces(indent_size), spaces())) + + # empty layout + self.assertEqual(ttk._format_layoutlist([])[0], '') + + # _format_layoutlist always expects the second item (in every item) + # to act like a dict (except when the value evaluates to False). + self.assertRaises(AttributeError, + ttk._format_layoutlist, [('a', 'b')]) + + smallest = ttk._format_layoutlist([('a', None)], indent=0) + self.assertEqual(smallest, + ttk._format_layoutlist([('a', '')], indent=0)) + self.assertEqual(smallest[0], 'a') + + # testing indentation levels + self.assertEqual(sample(), sample_expected()) + for i in range(4): + self.assertEqual(sample(i), sample_expected(i)) + self.assertEqual(sample(i, i), sample_expected(i, i)) + + # invalid layout format, different kind of exceptions will be + # raised by internal functions + + # plain wrong format + self.assertRaises(ValueError, ttk._format_layoutlist, + ['bad', 'format']) + # will try to use iteritems in the 'bad' string + self.assertRaises(AttributeError, ttk._format_layoutlist, + [('name', 'bad')]) + # bad children formatting + self.assertRaises(ValueError, ttk._format_layoutlist, + [('name', {'children': {'a': None}})]) + + + def test_script_from_settings(self): + # empty options + self.assertFalse(ttk._script_from_settings({'name': + {'configure': None, 'map': None, 'element create': None}})) + + # empty layout + self.assertEqual( + ttk._script_from_settings({'name': {'layout': None}}), + "ttk::style layout name {\nnull\n}") + + configdict = {'???': True, '?': False} + self.assertTrue( + ttk._script_from_settings({'name': {'configure': configdict}})) + + mapdict = {'?????d?': [('?', 'v?l')]} + self.assertTrue( + ttk._script_from_settings({'name': {'map': mapdict}})) + + # invalid image element + self.assertRaises(IndexError, + ttk._script_from_settings, {'name': {'element create': ['image']}}) + + # minimal valid image + self.assertTrue(ttk._script_from_settings({'name': + {'element create': ['image', 'name']}})) + + image = {'thing': {'element create': + ['image', 'name', ('state1', 'state2', 'val')]}} + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} ") + + image['thing']['element create'].append({'opt': 30}) + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt 30") + + image['thing']['element create'][-1]['opt'] = [MockTclObj(3), + MockTclObj('2m')] + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt {3 2m}") + + + def test_tclobj_to_py(self): + self.assertEqual( + ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), + [('a', 'b', 'val')]) + self.assertEqual( + ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), + [1, 2, '3m']) + + + def test_list_from_statespec(self): + def test_it(sspec, value, res_value, states): + self.assertEqual(ttk._list_from_statespec( + (sspec, value)), [states + (res_value, )]) + + states_even = tuple('state%d' % i for i in range(6)) + statespec = MockStateSpec(*states_even) + test_it(statespec, 'val', 'val', states_even) + test_it(statespec, MockTclObj('val'), 'val', states_even) + + states_odd = tuple('state%d' % i for i in range(5)) + statespec = MockStateSpec(*states_odd) + test_it(statespec, 'val', 'val', states_odd) + + test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) + + + def test_list_from_layouttuple(self): + tk = MockTkApp() + + # empty layout tuple + self.assertFalse(ttk._list_from_layouttuple(tk, ())) + + # shortest layout tuple + self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), + [('name', {})]) + + # not so interesting ltuple + sample_ltuple = ('name', '-option', 'value') + self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), + [('name', {'option': 'value'})]) + + # empty children + self.assertEqual(ttk._list_from_layouttuple(tk, + ('something', '-children', ())), + [('something', {'children': []})] + ) + + # more interesting ltuple + ltuple = ( + 'name', '-option', 'niceone', '-children', ( + ('otherone', '-children', ( + ('child', )), '-otheropt', 'othervalue' + ) + ) + ) + self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), + [('name', {'option': 'niceone', 'children': + [('otherone', {'otheropt': 'othervalue', 'children': + [('child', {})] + })] + })] + ) + + # bad tuples + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus', 'value')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('something', '-children')) # no children + + + def test_val_or_dict(self): + def func(res, opt=None, val=None): + if opt is None: + return res + if val is None: + return "test val" + return (opt, val) + + tk = MockTkApp() + tk.call = func + + self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), + {'test': '3'}) + self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), + {'test': 3}) + + self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), + 'test val') + + self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), + {'test': 3}) + + + def test_convert_stringval(self): + tests = ( + (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), + (None, 'None') + ) + for orig, expected in tests: + self.assertEqual(ttk._convert_stringval(orig), expected) + + +class TclObjsToPyTest(unittest.TestCase): + + def test_unicode(self): + adict = {'opt': 'v?l??'} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + adict['opt'] = MockTclObj(adict['opt']) + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + def test_multivalues(self): + adict = {'opt': [1, 2, 3, 4]} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) + + adict['opt'] = [1, 'xm', 3] + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) + + adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') + self.assertEqual(ttk.tclobjs_to_py(adict), + {'opt': [('a', 'b', 'v?l??')]}) + + self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), + {'x': ['y z']}) + + def test_nosplit(self): + self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), + {'text': 'some text'}) -def test_main(): - support.run_unittest( - *runtktests.get_tests(gui=False, packages=['test_ttk'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/tkinter/test/runtktests.py b/Lib/tkinter/test/runtktests.py deleted file mode 100644 index 33dc54a1375bf..0000000000000 --- a/Lib/tkinter/test/runtktests.py +++ /dev/null @@ -1,69 +0,0 @@ -""" -Use this module to get and run all tk tests. - -tkinter tests should live in a package inside the directory where this file -lives, like test_tkinter. -Extensions also should live in packages following the same rule as above. -""" - -import os -import importlib -import test.support - -this_dir_path = os.path.abspath(os.path.dirname(__file__)) - -def is_package(path): - for name in os.listdir(path): - if name in ('__init__.py', '__init__.pyc'): - return True - return False - -def get_tests_modules(basepath=this_dir_path, gui=True, packages=None): - """This will import and yield modules whose names start with test_ - and are inside packages found in the path starting at basepath. - - If packages is specified it should contain package names that - want their tests collected. - """ - py_ext = '.py' - - for dirpath, dirnames, filenames in os.walk(basepath): - for dirname in list(dirnames): - if dirname[0] == '.': - dirnames.remove(dirname) - - if is_package(dirpath) and filenames: - pkg_name = dirpath[len(basepath) + len(os.sep):].replace('/', '.') - if packages and pkg_name not in packages: - continue - - filenames = filter( - lambda x: x.startswith('test_') and x.endswith(py_ext), - filenames) - - for name in filenames: - try: - yield importlib.import_module( - ".%s.%s" % (pkg_name, name[:-len(py_ext)]), - "tkinter.test") - except test.support.ResourceDenied: - if gui: - raise - -def get_tests(text=True, gui=True, packages=None): - """Yield all the tests in the modules found by get_tests_modules. - - If nogui is True, only tests that do not require a GUI will be - returned.""" - attrs = [] - if text: - attrs.append('tests_nogui') - if gui: - attrs.append('tests_gui') - for module in get_tests_modules(gui=gui, packages=packages): - for attr in attrs: - for test in getattr(module, attr, ()): - yield test - -if __name__ == "__main__": - test.support.run_unittest(*get_tests()) diff --git a/Lib/tkinter/test/test_tkinter/test_colorchooser.py b/Lib/tkinter/test/test_tkinter/test_colorchooser.py index 41da86c2adef4..488162ff0dd96 100644 --- a/Lib/tkinter/test/test_tkinter/test_colorchooser.py +++ b/Lib/tkinter/test/test_tkinter/test_colorchooser.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest, AbstractTkTest from tkinter import colorchooser from tkinter.colorchooser import askcolor @@ -64,7 +64,5 @@ def test_callback(dialog, master): self.assertRaises(RuntimeError, askcolor) -tests_gui = (ChooserTest, DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_font.py b/Lib/tkinter/test/test_tkinter/test_font.py index 91f9974117197..058c53a902364 100644 --- a/Lib/tkinter/test/test_tkinter/test_font.py +++ b/Lib/tkinter/test/test_tkinter/test_font.py @@ -1,7 +1,7 @@ import unittest import tkinter from tkinter import font -from test.support import requires, run_unittest, gc_collect, ALWAYS_EQ +from test.support import requires, gc_collect, ALWAYS_EQ from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -159,7 +159,5 @@ def test_nametofont(self): self.assertRaises(RuntimeError, font.nametofont, fontname) -tests_gui = (FontTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_images.py b/Lib/tkinter/test/test_tkinter/test_images.py index c7b468044d55e..cc69ccac62d74 100644 --- a/Lib/tkinter/test/test_tkinter/test_images.py +++ b/Lib/tkinter/test/test_tkinter/test_images.py @@ -376,7 +376,5 @@ def test_transparency(self): self.assertEqual(image.transparency_get(4, 6), False) -tests_gui = (MiscTest, DefaultRootTest, BitmapImageTest, PhotoImageTest,) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_loadtk.py b/Lib/tkinter/test/test_tkinter/test_loadtk.py index 760ba72134082..61b0eda2fc750 100644 --- a/Lib/tkinter/test/test_tkinter/test_loadtk.py +++ b/Lib/tkinter/test/test_tkinter/test_loadtk.py @@ -41,7 +41,6 @@ def testLoadTkFailure(self): self.assertRaises(TclError, tcl.winfo_geometry) self.assertRaises(TclError, tcl.loadtk) -tests_gui = (TkLoadTest, ) if __name__ == "__main__": - test_support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_messagebox.py b/Lib/tkinter/test/test_tkinter/test_messagebox.py index 0dec08e9041a0..d38541a5a45e7 100644 --- a/Lib/tkinter/test/test_tkinter/test_messagebox.py +++ b/Lib/tkinter/test/test_tkinter/test_messagebox.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest from tkinter.commondialog import Dialog from tkinter.messagebox import showinfo @@ -32,7 +32,5 @@ def test_callback(dialog, master): self.assertRaises(RuntimeError, showinfo, "Spam", "Egg Information") -tests_gui = (DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index ab8f64790dfc0..2c78eaca2c85b 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -390,7 +390,5 @@ def test_mainloop(self): self.assertRaises(RuntimeError, tkinter.mainloop) -tests_gui = (MiscTest, DefaultRootTest) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_simpledialog.py b/Lib/tkinter/test/test_tkinter/test_simpledialog.py index b64b854c4db7e..18cd2712b0c5e 100644 --- a/Lib/tkinter/test/test_tkinter/test_simpledialog.py +++ b/Lib/tkinter/test/test_tkinter/test_simpledialog.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest from tkinter.simpledialog import Dialog, askinteger @@ -31,7 +31,5 @@ def mock_wait_window(w): self.assertRaises(RuntimeError, askinteger, "Go To Line", "Line number") -tests_gui = (DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_text.py b/Lib/tkinter/test/test_tkinter/test_text.py index 13b7c56a3978d..482f150df559f 100644 --- a/Lib/tkinter/test/test_tkinter/test_text.py +++ b/Lib/tkinter/test/test_tkinter/test_text.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -41,7 +41,5 @@ def test_search(self): self.assertEqual(text.search('test', '1.0', 'end'), '1.3') -tests_gui = (TextTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py index 0be5282a3a3b3..427e168454362 100644 --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -338,10 +338,5 @@ def test_variable(self): self.assertRaises(RuntimeError, Variable) -tests_gui = (TestVariable, TestStringVar, TestIntVar, - TestDoubleVar, TestBooleanVar, DefaultRootTest) - - if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index e6b3eccf7afb8..438d21d0b3733 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -2,7 +2,7 @@ import unittest import tkinter from tkinter import ttk -from test.support import requires, run_unittest, gc_collect +from test.support import requires, gc_collect from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -308,7 +308,5 @@ def test_labeledscale(self): self._test_widget(ttk.LabeledScale) -tests_gui = (LabeledScaleTest, OptionMenuTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py deleted file mode 100644 index 5c23d6fecf8b4..0000000000000 --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ /dev/null @@ -1,460 +0,0 @@ -# -*- encoding: utf-8 -*- -import unittest -from tkinter import ttk - -class MockTkApp: - - def splitlist(self, arg): - if isinstance(arg, tuple): - return arg - return arg.split(':') - - def wantobjects(self): - return True - - -class MockTclObj(object): - typename = 'test' - - def __init__(self, val): - self.val = val - - def __str__(self): - return str(self.val) - - -class MockStateSpec(object): - typename = 'StateSpec' - - def __init__(self, *args): - self.val = args - - def __str__(self): - return ' '.join(self.val) - - -class InternalFunctionsTest(unittest.TestCase): - - def test_format_optdict(self): - def check_against(fmt_opts, result): - for i in range(0, len(fmt_opts), 2): - self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) - if result: - self.fail("result still got elements: %s" % result) - - # passing an empty dict should return an empty object (tuple here) - self.assertFalse(ttk._format_optdict({})) - - # check list formatting - check_against( - ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), - {'-fg': 'blue', '-padding': '1 2 3 4'}) - - # check tuple formatting (same as list) - check_against( - ttk._format_optdict({'test': (1, 2, '', 0)}), - {'-test': '1 2 {} 0'}) - - # check untouched values - check_against( - ttk._format_optdict({'test': {'left': 'as is'}}), - {'-test': {'left': 'as is'}}) - - # check script formatting - check_against( - ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'test2': 3, - 'test3': '', 'test4': 'abc def', - 'test5': '"abc"', 'test6': '{}', - 'test7': '} -spam {'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-test2': '3', - '-test3': '{}', '-test4': '{abc def}', - '-test5': '{"abc"}', '-test6': r'\{\}', - '-test7': r'\}\ -spam\ \{'}) - - opts = {'???': True, '?': False} - orig_opts = opts.copy() - # check if giving unicode keys is fine - check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) - # opts should remain unchanged - self.assertEqual(opts, orig_opts) - - # passing values with spaces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one two', 'three')}), - {'-option': '{one two} three'}) - check_against( - ttk._format_optdict( - {'option': ('one\ttwo', 'three')}), - {'-option': '{one\ttwo} three'}) - - # passing empty strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('', 'one')}), - {'-option': '{} one'}) - - # passing values with braces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one} {two', 'three')}), - {'-option': r'one\}\ \{two three'}) - - # passing quoted strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('"one"', 'two')}), - {'-option': '{"one"} two'}) - check_against( - ttk._format_optdict( - {'option': ('{one}', 'two')}), - {'-option': r'\{one\} two'}) - - # ignore an option - amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore non-existing options - amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore every option - self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) - - - def test_format_mapdict(self): - opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} - result = ttk._format_mapdict(opts) - self.assertEqual(len(result), len(list(opts.keys())) * 2) - self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) - self.assertEqual(ttk._format_mapdict(opts, script=True), - ('-a', '{{b c} val d otherval {} single}')) - - self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) - - opts = {'?????d?': [('?', 'v?l')]} - result = ttk._format_mapdict(opts) - self.assertEqual(result, ('-?????d?', '? v?l')) - - self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), - ('-opt', '{} value')) - - # empty states - valid = {'opt': [('', '', 'hi')]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) - - # when passing multiple states, they all must be strings - invalid = {'opt': [(1, 2, 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - invalid = {'opt': [([1], '2', 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - # but when passing a single state, it can be anything - valid = {'opt': [[1, 'value']]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) - # special attention to single states which evaluate to False - for stateval in (None, 0, False, '', set()): # just some samples - valid = {'opt': [(stateval, 'value')]} - self.assertEqual(ttk._format_mapdict(valid), - ('-opt', '{} value')) - - # values must be iterable - opts = {'a': None} - self.assertRaises(TypeError, ttk._format_mapdict, opts) - - - def test_format_elemcreate(self): - self.assertTrue(ttk._format_elemcreate(None), (None, ())) - - ## Testing type = image - # image type expects at least an image name, so this should raise - # IndexError since it tries to access the index 0 of an empty tuple - self.assertRaises(IndexError, ttk._format_elemcreate, 'image') - - # don't format returned values as a tcl script - # minimum acceptable for image type - self.assertEqual(ttk._format_elemcreate('image', False, 'test'), - ("test ", ())) - # specifying a state spec - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('', 'a')), ("test {} a", ())) - # state spec with multiple states - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b', 'c')), ("test {a b} c", ())) - # state spec and options - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) - # format returned values as a tcl script - # state spec with multiple states and an option with a multivalue - self.assertEqual(ttk._format_elemcreate('image', True, 'test', - ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) - - ## Testing type = vsapi - # vsapi type expects at least a class name and a part_id, so this - # should raise a ValueError since it tries to get two elements from - # an empty tuple - self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') - - # don't format returned values as a tcl script - # minimum acceptable for vsapi - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), - ("a b ", ())) - # now with a state spec with multiple states - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b', 'c')), ("a b {a b} c", ())) - # state spec and option - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) - # format returned values as a tcl script - # state spec with a multivalue and an option - self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', - ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) - - # Testing type = from - # from type expects at least a type name - self.assertRaises(IndexError, ttk._format_elemcreate, 'from') - - self.assertEqual(ttk._format_elemcreate('from', False, 'a'), - ('a', ())) - self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), - ('a', ('b', ))) - self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), - ('{a}', 'b')) - - - def test_format_layoutlist(self): - def sample(indent=0, indent_size=2): - return ttk._format_layoutlist( - [('a', {'other': [1, 2, 3], 'children': - [('b', {'children': - [('c', {'children': - [('d', {'nice': 'opt'})], 'something': (1, 2) - })] - })] - })], indent=indent, indent_size=indent_size)[0] - - def sample_expected(indent=0, indent_size=2): - spaces = lambda amount=0: ' ' * (amount + indent) - return ( - "%sa -other {1 2 3} -children {\n" - "%sb -children {\n" - "%sc -something {1 2} -children {\n" - "%sd -nice opt\n" - "%s}\n" - "%s}\n" - "%s}" % (spaces(), spaces(indent_size), - spaces(2 * indent_size), spaces(3 * indent_size), - spaces(2 * indent_size), spaces(indent_size), spaces())) - - # empty layout - self.assertEqual(ttk._format_layoutlist([])[0], '') - - # _format_layoutlist always expects the second item (in every item) - # to act like a dict (except when the value evaluates to False). - self.assertRaises(AttributeError, - ttk._format_layoutlist, [('a', 'b')]) - - smallest = ttk._format_layoutlist([('a', None)], indent=0) - self.assertEqual(smallest, - ttk._format_layoutlist([('a', '')], indent=0)) - self.assertEqual(smallest[0], 'a') - - # testing indentation levels - self.assertEqual(sample(), sample_expected()) - for i in range(4): - self.assertEqual(sample(i), sample_expected(i)) - self.assertEqual(sample(i, i), sample_expected(i, i)) - - # invalid layout format, different kind of exceptions will be - # raised by internal functions - - # plain wrong format - self.assertRaises(ValueError, ttk._format_layoutlist, - ['bad', 'format']) - # will try to use iteritems in the 'bad' string - self.assertRaises(AttributeError, ttk._format_layoutlist, - [('name', 'bad')]) - # bad children formatting - self.assertRaises(ValueError, ttk._format_layoutlist, - [('name', {'children': {'a': None}})]) - - - def test_script_from_settings(self): - # empty options - self.assertFalse(ttk._script_from_settings({'name': - {'configure': None, 'map': None, 'element create': None}})) - - # empty layout - self.assertEqual( - ttk._script_from_settings({'name': {'layout': None}}), - "ttk::style layout name {\nnull\n}") - - configdict = {'???': True, '?': False} - self.assertTrue( - ttk._script_from_settings({'name': {'configure': configdict}})) - - mapdict = {'?????d?': [('?', 'v?l')]} - self.assertTrue( - ttk._script_from_settings({'name': {'map': mapdict}})) - - # invalid image element - self.assertRaises(IndexError, - ttk._script_from_settings, {'name': {'element create': ['image']}}) - - # minimal valid image - self.assertTrue(ttk._script_from_settings({'name': - {'element create': ['image', 'name']}})) - - image = {'thing': {'element create': - ['image', 'name', ('state1', 'state2', 'val')]}} - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} ") - - image['thing']['element create'].append({'opt': 30}) - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt 30") - - image['thing']['element create'][-1]['opt'] = [MockTclObj(3), - MockTclObj('2m')] - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt {3 2m}") - - - def test_tclobj_to_py(self): - self.assertEqual( - ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), - [('a', 'b', 'val')]) - self.assertEqual( - ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), - [1, 2, '3m']) - - - def test_list_from_statespec(self): - def test_it(sspec, value, res_value, states): - self.assertEqual(ttk._list_from_statespec( - (sspec, value)), [states + (res_value, )]) - - states_even = tuple('state%d' % i for i in range(6)) - statespec = MockStateSpec(*states_even) - test_it(statespec, 'val', 'val', states_even) - test_it(statespec, MockTclObj('val'), 'val', states_even) - - states_odd = tuple('state%d' % i for i in range(5)) - statespec = MockStateSpec(*states_odd) - test_it(statespec, 'val', 'val', states_odd) - - test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) - - - def test_list_from_layouttuple(self): - tk = MockTkApp() - - # empty layout tuple - self.assertFalse(ttk._list_from_layouttuple(tk, ())) - - # shortest layout tuple - self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), - [('name', {})]) - - # not so interesting ltuple - sample_ltuple = ('name', '-option', 'value') - self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), - [('name', {'option': 'value'})]) - - # empty children - self.assertEqual(ttk._list_from_layouttuple(tk, - ('something', '-children', ())), - [('something', {'children': []})] - ) - - # more interesting ltuple - ltuple = ( - 'name', '-option', 'niceone', '-children', ( - ('otherone', '-children', ( - ('child', )), '-otheropt', 'othervalue' - ) - ) - ) - self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), - [('name', {'option': 'niceone', 'children': - [('otherone', {'otheropt': 'othervalue', 'children': - [('child', {})] - })] - })] - ) - - # bad tuples - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus', 'value')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('something', '-children')) # no children - - - def test_val_or_dict(self): - def func(res, opt=None, val=None): - if opt is None: - return res - if val is None: - return "test val" - return (opt, val) - - tk = MockTkApp() - tk.call = func - - self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), - {'test': '3'}) - self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), - {'test': 3}) - - self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), - 'test val') - - self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), - {'test': 3}) - - - def test_convert_stringval(self): - tests = ( - (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), - (None, 'None') - ) - for orig, expected in tests: - self.assertEqual(ttk._convert_stringval(orig), expected) - - -class TclObjsToPyTest(unittest.TestCase): - - def test_unicode(self): - adict = {'opt': 'v?l??'} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - adict['opt'] = MockTclObj(adict['opt']) - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - def test_multivalues(self): - adict = {'opt': [1, 2, 3, 4]} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) - - adict['opt'] = [1, 'xm', 3] - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) - - adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') - self.assertEqual(ttk.tclobjs_to_py(adict), - {'opt': [('a', 'b', 'v?l??')]}) - - self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), - {'x': ['y z']}) - - def test_nosplit(self): - self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), - {'text': 'some text'}) - -tests_nogui = (InternalFunctionsTest, TclObjsToPyTest) - -if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_nogui) diff --git a/Lib/tkinter/test/test_ttk/test_style.py b/Lib/tkinter/test/test_ttk/test_style.py index 38d70d7a89077..a33c24ac55bee 100644 --- a/Lib/tkinter/test/test_ttk/test_style.py +++ b/Lib/tkinter/test/test_ttk/test_style.py @@ -3,7 +3,7 @@ import tkinter from tkinter import ttk from test import support -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -175,7 +175,5 @@ def test_map_custom_copy(self): self.assertEqual(style.map(newname, key), value) -tests_gui = (StyleTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 082da5d0c1a00..a2a4de2e5cc41 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -4,7 +4,7 @@ from test.support import requires, gc_collect import sys -from tkinter.test.test_ttk.test_functions import MockTclObj +from test.test_ttk_textonly import MockTclObj from tkinter.test.support import (AbstractTkTest, tcl_version, get_tk_patchlevel, simulate_mouse_click, AbstractDefaultRootTest) from tkinter.test.widget_tests import (add_standard_options, noconv, diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index ee225131854de..06e0a8b0d1c53 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -1424,7 +1424,6 @@ - @@ -1436,7 +1435,6 @@ - From webhook-mailer at python.org Wed Oct 13 12:21:32 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:21:32 -0000 Subject: [Python-checkins] bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) Message-ID: https://github.com/python/cpython/commit/b9e687618d3489944f29adbd2be50b46940c9e70 commit: b9e687618d3489944f29adbd2be50b46940c9e70 branch: main author: Ben Hoyt committer: ambv date: 2021-10-13T18:21:27+02:00 summary: bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst M Lib/email/_parseaddr.py M Lib/test/test_email/test_email.py diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py index 977fedf67b159..ba5ad5a36d06b 100644 --- a/Lib/email/_parseaddr.py +++ b/Lib/email/_parseaddr.py @@ -128,6 +128,8 @@ def _parsedate_tz(data): tss = 0 elif len(tm) == 3: [thh, tmm, tss] = tm + else: + return None else: return None try: diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 4001f716471dc..54ffcdc544e8b 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3009,6 +3009,7 @@ def test_parsedate_returns_None_for_invalid_strings(self): self.assertIsNone(utils.parsedate_tz('0')) self.assertIsNone(utils.parsedate('A Complete Waste of Time')) self.assertIsNone(utils.parsedate_tz('A Complete Waste of Time')) + self.assertIsNone(utils.parsedate_tz('Wed, 3 Apr 2002 12.34.56.78+0800')) # Not a part of the spec but, but this has historically worked: self.assertIsNone(utils.parsedate(None)) self.assertIsNone(utils.parsedate_tz(None)) diff --git a/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst new file mode 100644 index 0000000000000..9e5ec561c362a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst @@ -0,0 +1,3 @@ +Fixed :func:`email.utils.parsedate_tz` crashing with +:exc:`UnboundLocalError` on certain invalid input instead of returning +``None``. Patch by Ben Hoyt. From webhook-mailer at python.org Wed Oct 13 12:32:06 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:32:06 -0000 Subject: [Python-checkins] bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) Message-ID: https://github.com/python/cpython/commit/6fafc25aea8689048314b5bf7a9bb986bb1ce238 commit: 6fafc25aea8689048314b5bf7a9bb986bb1ce238 branch: main author: andrei kulakov committer: ambv date: 2021-10-13T18:31:51+02:00 summary: bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) files: A Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index df98999f875d9..4cc14f2372246 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -526,12 +526,13 @@ def _format_action(self, action): parts = [action_header] # if there was help for the action, add lines of help text - if action.help: + if action.help and action.help.strip(): help_text = self._expand_help(action) - help_lines = self._split_lines(help_text, help_width) - parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) - for line in help_lines[1:]: - parts.append('%*s%s\n' % (help_position, '', line)) + if help_text: + help_lines = self._split_lines(help_text, help_width) + parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) + for line in help_lines[1:]: + parts.append('%*s%s\n' % (help_position, '', line)) # or add a newline if the description doesn't end with one elif not action_header.endswith('\n'): diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 8babab399543b..21732d14604c0 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2164,6 +2164,42 @@ def test_help_non_breaking_spaces(self): wrap\N{NO-BREAK SPACE}at non-breaking spaces ''')) + def test_help_blank(self): + # Issue 24444 + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', + help=' ') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] foo + + main description + + positional arguments: + foo + + options: + -h, --help show this help message and exit + ''')) + + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', choices=[], + help='%(choices)s') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] {} + + main description + + positional arguments: + {} + + options: + -h, --help show this help message and exit + ''')) + def test_help_alternate_prefix_chars(self): parser = self._get_parser(prefix_chars='+:/') self.assertEqual(parser.format_usage(), diff --git a/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst new file mode 100644 index 0000000000000..efcacb8f0eb59 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst @@ -0,0 +1,2 @@ +Fixed an error raised in :mod:`argparse` help display when help for an +option is set to 1+ blank spaces or when *choices* arg is an empty container. From webhook-mailer at python.org Wed Oct 13 12:34:10 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:34:10 -0000 Subject: [Python-checkins] [docs] lexical_analysis: Expand the text on ``_`` (GH-28903) Message-ID: https://github.com/python/cpython/commit/3dee0cb6217db326e844955a7f8b424c67990557 commit: 3dee0cb6217db326e844955a7f8b424c67990557 branch: main author: Petr Viktorin committer: ambv date: 2021-10-13T18:34:01+02:00 summary: [docs] lexical_analysis: Expand the text on ``_`` (GH-28903) Also: * Expand the discussion into its own entry. (Even before this, text on ``_`` was longet than the text on ``_*``.) * Briefly note the other common convention for `_`: naming unused variables. Co-authored-by: Brandt Bucher Co-authored-by: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> Co-authored-by: ?ukasz Langa files: M Doc/reference/lexical_analysis.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 91ca8554adabf..4f7f00192d749 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -385,10 +385,20 @@ classes are identified by the patterns of leading and trailing underscore characters: ``_*`` - Not imported by ``from module import *``. The special identifier ``_`` is used - in the interactive interpreter to store the result of the last evaluation; it is - stored in the :mod:`builtins` module. When not in interactive mode, ``_`` - has no special meaning and is not defined. See section :ref:`import`. + Not imported by ``from module import *``. + +``_`` + In a ``case`` pattern within a :keyword:`match` statement, ``_`` is a + :ref:`soft keyword ` that denotes a + :ref:`wildcard `. + + Separately, the interactive interpreter makes the result of the last evaluation + available in the variable ``_``. + (It is stored in the :mod:`builtins` module, alongside built-in + functions like ``print``.) + + Elsewhere, ``_`` is a regular identifier. It is often used to name + "special" items, but it is not special to Python itself. .. note:: @@ -396,6 +406,8 @@ characters: refer to the documentation for the :mod:`gettext` module for more information on this convention. + It is also commonly used for unused variables. + ``__*__`` System-defined names, informally known as "dunder" names. These names are defined by the interpreter and its implementation (including the standard library). From webhook-mailer at python.org Wed Oct 13 12:38:41 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:38:41 -0000 Subject: [Python-checkins] bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) Message-ID: https://github.com/python/cpython/commit/1c831353816ff699b54e804047a7242a09e98f5b commit: 1c831353816ff699b54e804047a7242a09e98f5b branch: main author: rtobar committer: ambv date: 2021-10-13T18:38:36+02:00 summary: bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) At import time, the xmlrpc.client module uses different date formats to test strftime so it can format years with 4 digits consistently. Depending on the underlying C library and its strftime implementation some of these calls can result in ValueErrors, blocking the xmlrpc.client module from being imported. This commit changes the behavior of this bit of code to react to ValueError exceptions, treating the format that caused them as an non-viable option. files: A Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst M Lib/xmlrpc/client.py diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py index 9e7449c88dfc0..a614cef6ab2f1 100644 --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -264,16 +264,22 @@ def __repr__(self): # Issue #13305: different format codes across platforms _day0 = datetime(1, 1, 1) -if _day0.strftime('%Y') == '0001': # Mac OS X +def _try(fmt): + try: + return _day0.strftime(fmt) == '0001' + except ValueError: + return False +if _try('%Y'): # Mac OS X def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S") -elif _day0.strftime('%4Y') == '0001': # Linux +elif _try('%4Y'): # Linux def _iso8601_format(value): return value.strftime("%4Y%m%dT%H:%M:%S") else: def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S").zfill(17) del _day0 +del _try def _strftime(value): diff --git a/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst new file mode 100644 index 0000000000000..eec77ceccf933 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst @@ -0,0 +1,3 @@ +Make :mod:`xmlrpc.client` more robust to C runtimes where the underlying C +``strftime`` function results in a ``ValueError`` when testing for year +formatting options. From webhook-mailer at python.org Wed Oct 13 12:58:15 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:58:15 -0000 Subject: [Python-checkins] [docs] lexical_analysis: Expand the text on ``_`` (GH-28903) (GH-28933) Message-ID: https://github.com/python/cpython/commit/5abb2dec2cf147ce71cd15bcfda1bd095818f222 commit: 5abb2dec2cf147ce71cd15bcfda1bd095818f222 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T18:58:06+02:00 summary: [docs] lexical_analysis: Expand the text on ``_`` (GH-28903) (GH-28933) Also: * Expand the discussion into its own entry. (Even before this, text on ``_`` was longet than the text on ``_*``.) * Briefly note the other common convention for `_`: naming unused variables. Co-authored-by: Brandt Bucher Co-authored-by: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> Co-authored-by: ?ukasz Langa (cherry picked from commit 3dee0cb6217db326e844955a7f8b424c67990557) Co-authored-by: Petr Viktorin files: M Doc/reference/lexical_analysis.rst diff --git a/Doc/reference/lexical_analysis.rst b/Doc/reference/lexical_analysis.rst index 4ad8f8be1e7dd..daa746eea00c3 100644 --- a/Doc/reference/lexical_analysis.rst +++ b/Doc/reference/lexical_analysis.rst @@ -385,10 +385,20 @@ classes are identified by the patterns of leading and trailing underscore characters: ``_*`` - Not imported by ``from module import *``. The special identifier ``_`` is used - in the interactive interpreter to store the result of the last evaluation; it is - stored in the :mod:`builtins` module. When not in interactive mode, ``_`` - has no special meaning and is not defined. See section :ref:`import`. + Not imported by ``from module import *``. + +``_`` + In a ``case`` pattern within a :keyword:`match` statement, ``_`` is a + :ref:`soft keyword ` that denotes a + :ref:`wildcard `. + + Separately, the interactive interpreter makes the result of the last evaluation + available in the variable ``_``. + (It is stored in the :mod:`builtins` module, alongside built-in + functions like ``print``.) + + Elsewhere, ``_`` is a regular identifier. It is often used to name + "special" items, but it is not special to Python itself. .. note:: @@ -396,6 +406,8 @@ characters: refer to the documentation for the :mod:`gettext` module for more information on this convention. + It is also commonly used for unused variables. + ``__*__`` System-defined names, informally known as "dunder" names. These names are defined by the interpreter and its implementation (including the standard library). From webhook-mailer at python.org Wed Oct 13 12:58:42 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:58:42 -0000 Subject: [Python-checkins] bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) (GH-28928) Message-ID: https://github.com/python/cpython/commit/5638618845752f713c00d69dbe705fed16761948 commit: 5638618845752f713c00d69dbe705fed16761948 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T18:58:37+02:00 summary: bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) (GH-28928) Co-authored-by: ?ukasz Langa (cherry picked from commit b9e687618d3489944f29adbd2be50b46940c9e70) Co-authored-by: Ben Hoyt files: A Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst M Lib/email/_parseaddr.py M Lib/test/test_email/test_email.py diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py index 178329fbc6aac..c5a7b23193ef4 100644 --- a/Lib/email/_parseaddr.py +++ b/Lib/email/_parseaddr.py @@ -128,6 +128,8 @@ def _parsedate_tz(data): tss = 0 elif len(tm) == 3: [thh, tmm, tss] = tm + else: + return None else: return None try: diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 217a5b8710909..bc062b5cb0c59 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3008,6 +3008,7 @@ def test_parsedate_returns_None_for_invalid_strings(self): self.assertIsNone(utils.parsedate_tz('0')) self.assertIsNone(utils.parsedate('A Complete Waste of Time')) self.assertIsNone(utils.parsedate_tz('A Complete Waste of Time')) + self.assertIsNone(utils.parsedate_tz('Wed, 3 Apr 2002 12.34.56.78+0800')) # Not a part of the spec but, but this has historically worked: self.assertIsNone(utils.parsedate(None)) self.assertIsNone(utils.parsedate_tz(None)) diff --git a/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst new file mode 100644 index 0000000000000..9e5ec561c362a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst @@ -0,0 +1,3 @@ +Fixed :func:`email.utils.parsedate_tz` crashing with +:exc:`UnboundLocalError` on certain invalid input instead of returning +``None``. Patch by Ben Hoyt. From webhook-mailer at python.org Wed Oct 13 12:59:08 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 16:59:08 -0000 Subject: [Python-checkins] bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28927) Message-ID: https://github.com/python/cpython/commit/151234f5da4b1922e3d91233a6eddda92d7269b0 commit: 151234f5da4b1922e3d91233a6eddda92d7269b0 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T18:59:04+02:00 summary: bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28927) (cherry picked from commit f59ed3c310a7ceebf2a56a84ea969a7f75d95b64) Co-authored-by: Serhiy Storchaka files: D Lib/tkinter/test/runtktests.py D Lib/tkinter/test/test_ttk/test_functions.py M Lib/test/test_tk.py M Lib/test/test_ttk_guionly.py M Lib/test/test_ttk_textonly.py M Lib/tkinter/test/test_tkinter/test_colorchooser.py M Lib/tkinter/test/test_tkinter/test_font.py M Lib/tkinter/test/test_tkinter/test_images.py M Lib/tkinter/test/test_tkinter/test_loadtk.py M Lib/tkinter/test/test_tkinter/test_messagebox.py M Lib/tkinter/test/test_tkinter/test_misc.py M Lib/tkinter/test/test_tkinter/test_simpledialog.py M Lib/tkinter/test/test_tkinter/test_text.py M Lib/tkinter/test/test_tkinter/test_variables.py M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/test/test_ttk/test_style.py M Lib/tkinter/test/test_ttk/test_widgets.py M PCbuild/lib.pyproj diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py index 59842a5e25e36..69cc2322cd9aa 100644 --- a/Lib/test/test_tk.py +++ b/Lib/test/test_tk.py @@ -6,11 +6,9 @@ # Skip test if tk cannot be initialized. support.requires('gui') -from tkinter.test import runtktests +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_tkinter') -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_tkinter'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py index abb26433652f1..8f59839d066e6 100644 --- a/Lib/test/test_ttk_guionly.py +++ b/Lib/test/test_ttk_guionly.py @@ -11,25 +11,26 @@ import tkinter from _tkinter import TclError from tkinter import ttk -from tkinter.test import runtktests - -root = None -try: - root = tkinter.Tk() - button = ttk.Button(root) - button.destroy() - del button -except TclError as msg: - # assuming ttk is not available - raise unittest.SkipTest("ttk not available: %s" % msg) -finally: - if root is not None: - root.destroy() - del root - -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_ttk'])) + + +def setUpModule(): + root = None + try: + root = tkinter.Tk() + button = ttk.Button(root) + button.destroy() + del button + except TclError as msg: + # assuming ttk is not available + raise unittest.SkipTest("ttk not available: %s" % msg) + finally: + if root is not None: + root.destroy() + del root + +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_ttk') + if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_textonly.py b/Lib/test/test_ttk_textonly.py index 013b0e9ee21da..96dc179a69eca 100644 --- a/Lib/test/test_ttk_textonly.py +++ b/Lib/test/test_ttk_textonly.py @@ -1,15 +1,463 @@ -from test import support from test.support import import_helper - # Skip this test if _tkinter does not exist. import_helper.import_module('_tkinter') -from tkinter.test import runtktests +import unittest +from tkinter import ttk + + +class MockTkApp: + + def splitlist(self, arg): + if isinstance(arg, tuple): + return arg + return arg.split(':') + + def wantobjects(self): + return True + + +class MockTclObj(object): + typename = 'test' + + def __init__(self, val): + self.val = val + + def __str__(self): + return str(self.val) + + +class MockStateSpec(object): + typename = 'StateSpec' + + def __init__(self, *args): + self.val = args + + def __str__(self): + return ' '.join(self.val) + + +class InternalFunctionsTest(unittest.TestCase): + + def test_format_optdict(self): + def check_against(fmt_opts, result): + for i in range(0, len(fmt_opts), 2): + self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) + if result: + self.fail("result still got elements: %s" % result) + + # passing an empty dict should return an empty object (tuple here) + self.assertFalse(ttk._format_optdict({})) + + # check list formatting + check_against( + ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), + {'-fg': 'blue', '-padding': '1 2 3 4'}) + + # check tuple formatting (same as list) + check_against( + ttk._format_optdict({'test': (1, 2, '', 0)}), + {'-test': '1 2 {} 0'}) + + # check untouched values + check_against( + ttk._format_optdict({'test': {'left': 'as is'}}), + {'-test': {'left': 'as is'}}) + + # check script formatting + check_against( + ttk._format_optdict( + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) + + opts = {'???': True, '?': False} + orig_opts = opts.copy() + # check if giving unicode keys is fine + check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) + # opts should remain unchanged + self.assertEqual(opts, orig_opts) + + # passing values with spaces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one two', 'three')}), + {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) + + # ignore an option + amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore non-existing options + amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore every option + self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) + + + def test_format_mapdict(self): + opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} + result = ttk._format_mapdict(opts) + self.assertEqual(len(result), len(list(opts.keys())) * 2) + self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) + self.assertEqual(ttk._format_mapdict(opts, script=True), + ('-a', '{{b c} val d otherval {} single}')) + + self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) + + opts = {'?????d?': [('?', 'v?l')]} + result = ttk._format_mapdict(opts) + self.assertEqual(result, ('-?????d?', '? v?l')) + + self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), + ('-opt', '{} value')) + + # empty states + valid = {'opt': [('', '', 'hi')]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) + + # when passing multiple states, they all must be strings + invalid = {'opt': [(1, 2, 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + invalid = {'opt': [([1], '2', 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + # but when passing a single state, it can be anything + valid = {'opt': [[1, 'value']]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) + # special attention to single states which evaluate to False + for stateval in (None, 0, False, '', set()): # just some samples + valid = {'opt': [(stateval, 'value')]} + self.assertEqual(ttk._format_mapdict(valid), + ('-opt', '{} value')) + + # values must be iterable + opts = {'a': None} + self.assertRaises(TypeError, ttk._format_mapdict, opts) + + + def test_format_elemcreate(self): + self.assertTrue(ttk._format_elemcreate(None), (None, ())) + + ## Testing type = image + # image type expects at least an image name, so this should raise + # IndexError since it tries to access the index 0 of an empty tuple + self.assertRaises(IndexError, ttk._format_elemcreate, 'image') + + # don't format returned values as a tcl script + # minimum acceptable for image type + self.assertEqual(ttk._format_elemcreate('image', False, 'test'), + ("test ", ())) + # specifying a state spec + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('', 'a')), ("test {} a", ())) + # state spec with multiple states + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b', 'c')), ("test {a b} c", ())) + # state spec and options + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) + # format returned values as a tcl script + # state spec with multiple states and an option with a multivalue + self.assertEqual(ttk._format_elemcreate('image', True, 'test', + ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) + + ## Testing type = vsapi + # vsapi type expects at least a class name and a part_id, so this + # should raise a ValueError since it tries to get two elements from + # an empty tuple + self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') + + # don't format returned values as a tcl script + # minimum acceptable for vsapi + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), + ("a b ", ())) + # now with a state spec with multiple states + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b', 'c')), ("a b {a b} c", ())) + # state spec and option + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) + # format returned values as a tcl script + # state spec with a multivalue and an option + self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', + ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) + + # Testing type = from + # from type expects at least a type name + self.assertRaises(IndexError, ttk._format_elemcreate, 'from') + + self.assertEqual(ttk._format_elemcreate('from', False, 'a'), + ('a', ())) + self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), + ('a', ('b', ))) + self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), + ('{a}', 'b')) + + + def test_format_layoutlist(self): + def sample(indent=0, indent_size=2): + return ttk._format_layoutlist( + [('a', {'other': [1, 2, 3], 'children': + [('b', {'children': + [('c', {'children': + [('d', {'nice': 'opt'})], 'something': (1, 2) + })] + })] + })], indent=indent, indent_size=indent_size)[0] + + def sample_expected(indent=0, indent_size=2): + spaces = lambda amount=0: ' ' * (amount + indent) + return ( + "%sa -other {1 2 3} -children {\n" + "%sb -children {\n" + "%sc -something {1 2} -children {\n" + "%sd -nice opt\n" + "%s}\n" + "%s}\n" + "%s}" % (spaces(), spaces(indent_size), + spaces(2 * indent_size), spaces(3 * indent_size), + spaces(2 * indent_size), spaces(indent_size), spaces())) + + # empty layout + self.assertEqual(ttk._format_layoutlist([])[0], '') + + # _format_layoutlist always expects the second item (in every item) + # to act like a dict (except when the value evaluates to False). + self.assertRaises(AttributeError, + ttk._format_layoutlist, [('a', 'b')]) + + smallest = ttk._format_layoutlist([('a', None)], indent=0) + self.assertEqual(smallest, + ttk._format_layoutlist([('a', '')], indent=0)) + self.assertEqual(smallest[0], 'a') + + # testing indentation levels + self.assertEqual(sample(), sample_expected()) + for i in range(4): + self.assertEqual(sample(i), sample_expected(i)) + self.assertEqual(sample(i, i), sample_expected(i, i)) + + # invalid layout format, different kind of exceptions will be + # raised by internal functions + + # plain wrong format + self.assertRaises(ValueError, ttk._format_layoutlist, + ['bad', 'format']) + # will try to use iteritems in the 'bad' string + self.assertRaises(AttributeError, ttk._format_layoutlist, + [('name', 'bad')]) + # bad children formatting + self.assertRaises(ValueError, ttk._format_layoutlist, + [('name', {'children': {'a': None}})]) + + + def test_script_from_settings(self): + # empty options + self.assertFalse(ttk._script_from_settings({'name': + {'configure': None, 'map': None, 'element create': None}})) + + # empty layout + self.assertEqual( + ttk._script_from_settings({'name': {'layout': None}}), + "ttk::style layout name {\nnull\n}") + + configdict = {'???': True, '?': False} + self.assertTrue( + ttk._script_from_settings({'name': {'configure': configdict}})) + + mapdict = {'?????d?': [('?', 'v?l')]} + self.assertTrue( + ttk._script_from_settings({'name': {'map': mapdict}})) + + # invalid image element + self.assertRaises(IndexError, + ttk._script_from_settings, {'name': {'element create': ['image']}}) + + # minimal valid image + self.assertTrue(ttk._script_from_settings({'name': + {'element create': ['image', 'name']}})) + + image = {'thing': {'element create': + ['image', 'name', ('state1', 'state2', 'val')]}} + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} ") + + image['thing']['element create'].append({'opt': 30}) + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt 30") + + image['thing']['element create'][-1]['opt'] = [MockTclObj(3), + MockTclObj('2m')] + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt {3 2m}") + + + def test_tclobj_to_py(self): + self.assertEqual( + ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), + [('a', 'b', 'val')]) + self.assertEqual( + ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), + [1, 2, '3m']) + + + def test_list_from_statespec(self): + def test_it(sspec, value, res_value, states): + self.assertEqual(ttk._list_from_statespec( + (sspec, value)), [states + (res_value, )]) + + states_even = tuple('state%d' % i for i in range(6)) + statespec = MockStateSpec(*states_even) + test_it(statespec, 'val', 'val', states_even) + test_it(statespec, MockTclObj('val'), 'val', states_even) + + states_odd = tuple('state%d' % i for i in range(5)) + statespec = MockStateSpec(*states_odd) + test_it(statespec, 'val', 'val', states_odd) + + test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) + + + def test_list_from_layouttuple(self): + tk = MockTkApp() + + # empty layout tuple + self.assertFalse(ttk._list_from_layouttuple(tk, ())) + + # shortest layout tuple + self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), + [('name', {})]) + + # not so interesting ltuple + sample_ltuple = ('name', '-option', 'value') + self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), + [('name', {'option': 'value'})]) + + # empty children + self.assertEqual(ttk._list_from_layouttuple(tk, + ('something', '-children', ())), + [('something', {'children': []})] + ) + + # more interesting ltuple + ltuple = ( + 'name', '-option', 'niceone', '-children', ( + ('otherone', '-children', ( + ('child', )), '-otheropt', 'othervalue' + ) + ) + ) + self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), + [('name', {'option': 'niceone', 'children': + [('otherone', {'otheropt': 'othervalue', 'children': + [('child', {})] + })] + })] + ) + + # bad tuples + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus', 'value')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('something', '-children')) # no children + + + def test_val_or_dict(self): + def func(res, opt=None, val=None): + if opt is None: + return res + if val is None: + return "test val" + return (opt, val) + + tk = MockTkApp() + tk.call = func + + self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), + {'test': '3'}) + self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), + {'test': 3}) + + self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), + 'test val') + + self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), + {'test': 3}) + + + def test_convert_stringval(self): + tests = ( + (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), + (None, 'None') + ) + for orig, expected in tests: + self.assertEqual(ttk._convert_stringval(orig), expected) + + +class TclObjsToPyTest(unittest.TestCase): + + def test_unicode(self): + adict = {'opt': 'v?l??'} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + adict['opt'] = MockTclObj(adict['opt']) + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + def test_multivalues(self): + adict = {'opt': [1, 2, 3, 4]} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) + + adict['opt'] = [1, 'xm', 3] + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) + + adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') + self.assertEqual(ttk.tclobjs_to_py(adict), + {'opt': [('a', 'b', 'v?l??')]}) + + self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), + {'x': ['y z']}) + + def test_nosplit(self): + self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), + {'text': 'some text'}) -def test_main(): - support.run_unittest( - *runtktests.get_tests(gui=False, packages=['test_ttk'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/tkinter/test/runtktests.py b/Lib/tkinter/test/runtktests.py deleted file mode 100644 index 33dc54a1375bf..0000000000000 --- a/Lib/tkinter/test/runtktests.py +++ /dev/null @@ -1,69 +0,0 @@ -""" -Use this module to get and run all tk tests. - -tkinter tests should live in a package inside the directory where this file -lives, like test_tkinter. -Extensions also should live in packages following the same rule as above. -""" - -import os -import importlib -import test.support - -this_dir_path = os.path.abspath(os.path.dirname(__file__)) - -def is_package(path): - for name in os.listdir(path): - if name in ('__init__.py', '__init__.pyc'): - return True - return False - -def get_tests_modules(basepath=this_dir_path, gui=True, packages=None): - """This will import and yield modules whose names start with test_ - and are inside packages found in the path starting at basepath. - - If packages is specified it should contain package names that - want their tests collected. - """ - py_ext = '.py' - - for dirpath, dirnames, filenames in os.walk(basepath): - for dirname in list(dirnames): - if dirname[0] == '.': - dirnames.remove(dirname) - - if is_package(dirpath) and filenames: - pkg_name = dirpath[len(basepath) + len(os.sep):].replace('/', '.') - if packages and pkg_name not in packages: - continue - - filenames = filter( - lambda x: x.startswith('test_') and x.endswith(py_ext), - filenames) - - for name in filenames: - try: - yield importlib.import_module( - ".%s.%s" % (pkg_name, name[:-len(py_ext)]), - "tkinter.test") - except test.support.ResourceDenied: - if gui: - raise - -def get_tests(text=True, gui=True, packages=None): - """Yield all the tests in the modules found by get_tests_modules. - - If nogui is True, only tests that do not require a GUI will be - returned.""" - attrs = [] - if text: - attrs.append('tests_nogui') - if gui: - attrs.append('tests_gui') - for module in get_tests_modules(gui=gui, packages=packages): - for attr in attrs: - for test in getattr(module, attr, ()): - yield test - -if __name__ == "__main__": - test.support.run_unittest(*get_tests()) diff --git a/Lib/tkinter/test/test_tkinter/test_colorchooser.py b/Lib/tkinter/test/test_tkinter/test_colorchooser.py index 41da86c2adef4..488162ff0dd96 100644 --- a/Lib/tkinter/test/test_tkinter/test_colorchooser.py +++ b/Lib/tkinter/test/test_tkinter/test_colorchooser.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest, AbstractTkTest from tkinter import colorchooser from tkinter.colorchooser import askcolor @@ -64,7 +64,5 @@ def test_callback(dialog, master): self.assertRaises(RuntimeError, askcolor) -tests_gui = (ChooserTest, DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_font.py b/Lib/tkinter/test/test_tkinter/test_font.py index 91f9974117197..058c53a902364 100644 --- a/Lib/tkinter/test/test_tkinter/test_font.py +++ b/Lib/tkinter/test/test_tkinter/test_font.py @@ -1,7 +1,7 @@ import unittest import tkinter from tkinter import font -from test.support import requires, run_unittest, gc_collect, ALWAYS_EQ +from test.support import requires, gc_collect, ALWAYS_EQ from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -159,7 +159,5 @@ def test_nametofont(self): self.assertRaises(RuntimeError, font.nametofont, fontname) -tests_gui = (FontTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_images.py b/Lib/tkinter/test/test_tkinter/test_images.py index c7b468044d55e..cc69ccac62d74 100644 --- a/Lib/tkinter/test/test_tkinter/test_images.py +++ b/Lib/tkinter/test/test_tkinter/test_images.py @@ -376,7 +376,5 @@ def test_transparency(self): self.assertEqual(image.transparency_get(4, 6), False) -tests_gui = (MiscTest, DefaultRootTest, BitmapImageTest, PhotoImageTest,) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_loadtk.py b/Lib/tkinter/test/test_tkinter/test_loadtk.py index 760ba72134082..61b0eda2fc750 100644 --- a/Lib/tkinter/test/test_tkinter/test_loadtk.py +++ b/Lib/tkinter/test/test_tkinter/test_loadtk.py @@ -41,7 +41,6 @@ def testLoadTkFailure(self): self.assertRaises(TclError, tcl.winfo_geometry) self.assertRaises(TclError, tcl.loadtk) -tests_gui = (TkLoadTest, ) if __name__ == "__main__": - test_support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_messagebox.py b/Lib/tkinter/test/test_tkinter/test_messagebox.py index 0dec08e9041a0..d38541a5a45e7 100644 --- a/Lib/tkinter/test/test_tkinter/test_messagebox.py +++ b/Lib/tkinter/test/test_tkinter/test_messagebox.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest from tkinter.commondialog import Dialog from tkinter.messagebox import showinfo @@ -32,7 +32,5 @@ def test_callback(dialog, master): self.assertRaises(RuntimeError, showinfo, "Spam", "Egg Information") -tests_gui = (DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 7e78433a7c9ac..b9201f1070940 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -346,7 +346,5 @@ def test_mainloop(self): self.assertRaises(RuntimeError, tkinter.mainloop) -tests_gui = (MiscTest, DefaultRootTest) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_simpledialog.py b/Lib/tkinter/test/test_tkinter/test_simpledialog.py index b64b854c4db7e..18cd2712b0c5e 100644 --- a/Lib/tkinter/test/test_tkinter/test_simpledialog.py +++ b/Lib/tkinter/test/test_tkinter/test_simpledialog.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest from tkinter.simpledialog import Dialog, askinteger @@ -31,7 +31,5 @@ def mock_wait_window(w): self.assertRaises(RuntimeError, askinteger, "Go To Line", "Line number") -tests_gui = (DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_text.py b/Lib/tkinter/test/test_tkinter/test_text.py index 13b7c56a3978d..482f150df559f 100644 --- a/Lib/tkinter/test/test_tkinter/test_text.py +++ b/Lib/tkinter/test/test_tkinter/test_text.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -41,7 +41,5 @@ def test_search(self): self.assertEqual(text.search('test', '1.0', 'end'), '1.3') -tests_gui = (TextTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py index 0be5282a3a3b3..427e168454362 100644 --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -338,10 +338,5 @@ def test_variable(self): self.assertRaises(RuntimeError, Variable) -tests_gui = (TestVariable, TestStringVar, TestIntVar, - TestDoubleVar, TestBooleanVar, DefaultRootTest) - - if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index e6b3eccf7afb8..438d21d0b3733 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -2,7 +2,7 @@ import unittest import tkinter from tkinter import ttk -from test.support import requires, run_unittest, gc_collect +from test.support import requires, gc_collect from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -308,7 +308,5 @@ def test_labeledscale(self): self._test_widget(ttk.LabeledScale) -tests_gui = (LabeledScaleTest, OptionMenuTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py deleted file mode 100644 index 5c23d6fecf8b4..0000000000000 --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ /dev/null @@ -1,460 +0,0 @@ -# -*- encoding: utf-8 -*- -import unittest -from tkinter import ttk - -class MockTkApp: - - def splitlist(self, arg): - if isinstance(arg, tuple): - return arg - return arg.split(':') - - def wantobjects(self): - return True - - -class MockTclObj(object): - typename = 'test' - - def __init__(self, val): - self.val = val - - def __str__(self): - return str(self.val) - - -class MockStateSpec(object): - typename = 'StateSpec' - - def __init__(self, *args): - self.val = args - - def __str__(self): - return ' '.join(self.val) - - -class InternalFunctionsTest(unittest.TestCase): - - def test_format_optdict(self): - def check_against(fmt_opts, result): - for i in range(0, len(fmt_opts), 2): - self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) - if result: - self.fail("result still got elements: %s" % result) - - # passing an empty dict should return an empty object (tuple here) - self.assertFalse(ttk._format_optdict({})) - - # check list formatting - check_against( - ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), - {'-fg': 'blue', '-padding': '1 2 3 4'}) - - # check tuple formatting (same as list) - check_against( - ttk._format_optdict({'test': (1, 2, '', 0)}), - {'-test': '1 2 {} 0'}) - - # check untouched values - check_against( - ttk._format_optdict({'test': {'left': 'as is'}}), - {'-test': {'left': 'as is'}}) - - # check script formatting - check_against( - ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'test2': 3, - 'test3': '', 'test4': 'abc def', - 'test5': '"abc"', 'test6': '{}', - 'test7': '} -spam {'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-test2': '3', - '-test3': '{}', '-test4': '{abc def}', - '-test5': '{"abc"}', '-test6': r'\{\}', - '-test7': r'\}\ -spam\ \{'}) - - opts = {'???': True, '?': False} - orig_opts = opts.copy() - # check if giving unicode keys is fine - check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) - # opts should remain unchanged - self.assertEqual(opts, orig_opts) - - # passing values with spaces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one two', 'three')}), - {'-option': '{one two} three'}) - check_against( - ttk._format_optdict( - {'option': ('one\ttwo', 'three')}), - {'-option': '{one\ttwo} three'}) - - # passing empty strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('', 'one')}), - {'-option': '{} one'}) - - # passing values with braces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one} {two', 'three')}), - {'-option': r'one\}\ \{two three'}) - - # passing quoted strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('"one"', 'two')}), - {'-option': '{"one"} two'}) - check_against( - ttk._format_optdict( - {'option': ('{one}', 'two')}), - {'-option': r'\{one\} two'}) - - # ignore an option - amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore non-existing options - amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore every option - self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) - - - def test_format_mapdict(self): - opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} - result = ttk._format_mapdict(opts) - self.assertEqual(len(result), len(list(opts.keys())) * 2) - self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) - self.assertEqual(ttk._format_mapdict(opts, script=True), - ('-a', '{{b c} val d otherval {} single}')) - - self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) - - opts = {'?????d?': [('?', 'v?l')]} - result = ttk._format_mapdict(opts) - self.assertEqual(result, ('-?????d?', '? v?l')) - - self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), - ('-opt', '{} value')) - - # empty states - valid = {'opt': [('', '', 'hi')]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) - - # when passing multiple states, they all must be strings - invalid = {'opt': [(1, 2, 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - invalid = {'opt': [([1], '2', 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - # but when passing a single state, it can be anything - valid = {'opt': [[1, 'value']]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) - # special attention to single states which evaluate to False - for stateval in (None, 0, False, '', set()): # just some samples - valid = {'opt': [(stateval, 'value')]} - self.assertEqual(ttk._format_mapdict(valid), - ('-opt', '{} value')) - - # values must be iterable - opts = {'a': None} - self.assertRaises(TypeError, ttk._format_mapdict, opts) - - - def test_format_elemcreate(self): - self.assertTrue(ttk._format_elemcreate(None), (None, ())) - - ## Testing type = image - # image type expects at least an image name, so this should raise - # IndexError since it tries to access the index 0 of an empty tuple - self.assertRaises(IndexError, ttk._format_elemcreate, 'image') - - # don't format returned values as a tcl script - # minimum acceptable for image type - self.assertEqual(ttk._format_elemcreate('image', False, 'test'), - ("test ", ())) - # specifying a state spec - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('', 'a')), ("test {} a", ())) - # state spec with multiple states - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b', 'c')), ("test {a b} c", ())) - # state spec and options - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) - # format returned values as a tcl script - # state spec with multiple states and an option with a multivalue - self.assertEqual(ttk._format_elemcreate('image', True, 'test', - ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) - - ## Testing type = vsapi - # vsapi type expects at least a class name and a part_id, so this - # should raise a ValueError since it tries to get two elements from - # an empty tuple - self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') - - # don't format returned values as a tcl script - # minimum acceptable for vsapi - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), - ("a b ", ())) - # now with a state spec with multiple states - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b', 'c')), ("a b {a b} c", ())) - # state spec and option - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) - # format returned values as a tcl script - # state spec with a multivalue and an option - self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', - ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) - - # Testing type = from - # from type expects at least a type name - self.assertRaises(IndexError, ttk._format_elemcreate, 'from') - - self.assertEqual(ttk._format_elemcreate('from', False, 'a'), - ('a', ())) - self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), - ('a', ('b', ))) - self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), - ('{a}', 'b')) - - - def test_format_layoutlist(self): - def sample(indent=0, indent_size=2): - return ttk._format_layoutlist( - [('a', {'other': [1, 2, 3], 'children': - [('b', {'children': - [('c', {'children': - [('d', {'nice': 'opt'})], 'something': (1, 2) - })] - })] - })], indent=indent, indent_size=indent_size)[0] - - def sample_expected(indent=0, indent_size=2): - spaces = lambda amount=0: ' ' * (amount + indent) - return ( - "%sa -other {1 2 3} -children {\n" - "%sb -children {\n" - "%sc -something {1 2} -children {\n" - "%sd -nice opt\n" - "%s}\n" - "%s}\n" - "%s}" % (spaces(), spaces(indent_size), - spaces(2 * indent_size), spaces(3 * indent_size), - spaces(2 * indent_size), spaces(indent_size), spaces())) - - # empty layout - self.assertEqual(ttk._format_layoutlist([])[0], '') - - # _format_layoutlist always expects the second item (in every item) - # to act like a dict (except when the value evaluates to False). - self.assertRaises(AttributeError, - ttk._format_layoutlist, [('a', 'b')]) - - smallest = ttk._format_layoutlist([('a', None)], indent=0) - self.assertEqual(smallest, - ttk._format_layoutlist([('a', '')], indent=0)) - self.assertEqual(smallest[0], 'a') - - # testing indentation levels - self.assertEqual(sample(), sample_expected()) - for i in range(4): - self.assertEqual(sample(i), sample_expected(i)) - self.assertEqual(sample(i, i), sample_expected(i, i)) - - # invalid layout format, different kind of exceptions will be - # raised by internal functions - - # plain wrong format - self.assertRaises(ValueError, ttk._format_layoutlist, - ['bad', 'format']) - # will try to use iteritems in the 'bad' string - self.assertRaises(AttributeError, ttk._format_layoutlist, - [('name', 'bad')]) - # bad children formatting - self.assertRaises(ValueError, ttk._format_layoutlist, - [('name', {'children': {'a': None}})]) - - - def test_script_from_settings(self): - # empty options - self.assertFalse(ttk._script_from_settings({'name': - {'configure': None, 'map': None, 'element create': None}})) - - # empty layout - self.assertEqual( - ttk._script_from_settings({'name': {'layout': None}}), - "ttk::style layout name {\nnull\n}") - - configdict = {'???': True, '?': False} - self.assertTrue( - ttk._script_from_settings({'name': {'configure': configdict}})) - - mapdict = {'?????d?': [('?', 'v?l')]} - self.assertTrue( - ttk._script_from_settings({'name': {'map': mapdict}})) - - # invalid image element - self.assertRaises(IndexError, - ttk._script_from_settings, {'name': {'element create': ['image']}}) - - # minimal valid image - self.assertTrue(ttk._script_from_settings({'name': - {'element create': ['image', 'name']}})) - - image = {'thing': {'element create': - ['image', 'name', ('state1', 'state2', 'val')]}} - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} ") - - image['thing']['element create'].append({'opt': 30}) - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt 30") - - image['thing']['element create'][-1]['opt'] = [MockTclObj(3), - MockTclObj('2m')] - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt {3 2m}") - - - def test_tclobj_to_py(self): - self.assertEqual( - ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), - [('a', 'b', 'val')]) - self.assertEqual( - ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), - [1, 2, '3m']) - - - def test_list_from_statespec(self): - def test_it(sspec, value, res_value, states): - self.assertEqual(ttk._list_from_statespec( - (sspec, value)), [states + (res_value, )]) - - states_even = tuple('state%d' % i for i in range(6)) - statespec = MockStateSpec(*states_even) - test_it(statespec, 'val', 'val', states_even) - test_it(statespec, MockTclObj('val'), 'val', states_even) - - states_odd = tuple('state%d' % i for i in range(5)) - statespec = MockStateSpec(*states_odd) - test_it(statespec, 'val', 'val', states_odd) - - test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) - - - def test_list_from_layouttuple(self): - tk = MockTkApp() - - # empty layout tuple - self.assertFalse(ttk._list_from_layouttuple(tk, ())) - - # shortest layout tuple - self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), - [('name', {})]) - - # not so interesting ltuple - sample_ltuple = ('name', '-option', 'value') - self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), - [('name', {'option': 'value'})]) - - # empty children - self.assertEqual(ttk._list_from_layouttuple(tk, - ('something', '-children', ())), - [('something', {'children': []})] - ) - - # more interesting ltuple - ltuple = ( - 'name', '-option', 'niceone', '-children', ( - ('otherone', '-children', ( - ('child', )), '-otheropt', 'othervalue' - ) - ) - ) - self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), - [('name', {'option': 'niceone', 'children': - [('otherone', {'otheropt': 'othervalue', 'children': - [('child', {})] - })] - })] - ) - - # bad tuples - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus', 'value')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('something', '-children')) # no children - - - def test_val_or_dict(self): - def func(res, opt=None, val=None): - if opt is None: - return res - if val is None: - return "test val" - return (opt, val) - - tk = MockTkApp() - tk.call = func - - self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), - {'test': '3'}) - self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), - {'test': 3}) - - self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), - 'test val') - - self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), - {'test': 3}) - - - def test_convert_stringval(self): - tests = ( - (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), - (None, 'None') - ) - for orig, expected in tests: - self.assertEqual(ttk._convert_stringval(orig), expected) - - -class TclObjsToPyTest(unittest.TestCase): - - def test_unicode(self): - adict = {'opt': 'v?l??'} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - adict['opt'] = MockTclObj(adict['opt']) - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - def test_multivalues(self): - adict = {'opt': [1, 2, 3, 4]} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) - - adict['opt'] = [1, 'xm', 3] - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) - - adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') - self.assertEqual(ttk.tclobjs_to_py(adict), - {'opt': [('a', 'b', 'v?l??')]}) - - self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), - {'x': ['y z']}) - - def test_nosplit(self): - self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), - {'text': 'some text'}) - -tests_nogui = (InternalFunctionsTest, TclObjsToPyTest) - -if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_nogui) diff --git a/Lib/tkinter/test/test_ttk/test_style.py b/Lib/tkinter/test/test_ttk/test_style.py index 38d70d7a89077..a33c24ac55bee 100644 --- a/Lib/tkinter/test/test_ttk/test_style.py +++ b/Lib/tkinter/test/test_ttk/test_style.py @@ -3,7 +3,7 @@ import tkinter from tkinter import ttk from test import support -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -175,7 +175,5 @@ def test_map_custom_copy(self): self.assertEqual(style.map(newname, key), value) -tests_gui = (StyleTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 082da5d0c1a00..a2a4de2e5cc41 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -4,7 +4,7 @@ from test.support import requires, gc_collect import sys -from tkinter.test.test_ttk.test_functions import MockTclObj +from test.test_ttk_textonly import MockTclObj from tkinter.test.support import (AbstractTkTest, tcl_version, get_tk_patchlevel, simulate_mouse_click, AbstractDefaultRootTest) from tkinter.test.widget_tests import (add_standard_options, noconv, diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index eba4d7591568d..c826ffa080c50 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -1429,7 +1429,6 @@ - @@ -1441,7 +1440,6 @@ - From webhook-mailer at python.org Wed Oct 13 13:08:27 2021 From: webhook-mailer at python.org (pablogsal) Date: Wed, 13 Oct 2021 17:08:27 -0000 Subject: [Python-checkins] bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823) Message-ID: https://github.com/python/cpython/commit/db2b6a20cd35781b2f5e798e880e57e6cf9b97aa commit: db2b6a20cd35781b2f5e798e880e57e6cf9b97aa branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-13T18:08:19+01:00 summary: bpo-45445: Fail if an invalid X-option is provided in the command line (GH-28823) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-12-14-41-39.bpo-45445._F5cMf.rst A test_foo.py M Doc/library/sys.rst M Lib/test/test_audit.py M Lib/test/test_cmd_line.py M Lib/test/test_embed.py M Programs/_testembed.c M Python/initconfig.c diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 8b3c6fd762731..ee07ba1042f62 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -1729,13 +1729,13 @@ always available. .. code-block:: shell-session - $ ./python -Xa=b -Xc + $ ./python -Xpycache_prefix=some_path -Xdev Python 3.2a3+ (py3k, Oct 16 2010, 20:14:50) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys._xoptions - {'a': 'b', 'c': True} + {'pycache_prefix': 'some_path', 'dev': True} .. impl-detail:: diff --git a/Lib/test/test_audit.py b/Lib/test/test_audit.py index c5ce26323b5f9..d99b3b7ed7d36 100644 --- a/Lib/test/test_audit.py +++ b/Lib/test/test_audit.py @@ -18,7 +18,7 @@ class AuditTest(unittest.TestCase): def do_test(self, *args): with subprocess.Popen( - [sys.executable, "-X utf8", AUDIT_TESTS_PY, *args], + [sys.executable, "-Xutf8", AUDIT_TESTS_PY, *args], encoding="utf-8", stdout=subprocess.PIPE, stderr=subprocess.PIPE, @@ -32,7 +32,7 @@ def do_test(self, *args): def run_python(self, *args): events = [] with subprocess.Popen( - [sys.executable, "-X utf8", AUDIT_TESTS_PY, *args], + [sys.executable, "-Xutf8", AUDIT_TESTS_PY, *args], encoding="utf-8", stdout=subprocess.PIPE, stderr=subprocess.PIPE, diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py index d93e98f372532..1dc8c45885cbe 100644 --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -83,8 +83,17 @@ def get_xoptions(*args): opts = get_xoptions() self.assertEqual(opts, {}) - opts = get_xoptions('-Xa', '-Xb=c,d=e') - self.assertEqual(opts, {'a': True, 'b': 'c,d=e'}) + opts = get_xoptions('-Xno_debug_ranges', '-Xdev=1234') + self.assertEqual(opts, {'no_debug_ranges': True, 'dev': '1234'}) + + @unittest.skipIf(interpreter_requires_environment(), + 'Cannot run -E tests when PYTHON env vars are required.') + def test_unknown_xoptions(self): + rc, out, err = assert_python_failure('-X', 'blech') + self.assertIn(b'Unknown value for option -X', err) + msg = b'Fatal Python error: Unknown value for option -X' + self.assertEqual(err.splitlines().count(msg), 1) + self.assertEqual(b'', out) def test_showrefcount(self): def run_python(*args): diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 31c5c3e49dda1..41e092019c49a 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -273,7 +273,7 @@ def test_pre_initialization_sys_options(self): "test_pre_initialization_sys_options", env=env) expected_output = ( "sys.warnoptions: ['once', 'module', 'default']\n" - "sys._xoptions: {'not_an_option': '1', 'also_not_an_option': '2'}\n" + "sys._xoptions: {'dev': '2', 'utf8': '1'}\n" "warnings.filters[:3]: ['default', 'module', 'once']\n" ) self.assertIn(expected_output, out) @@ -820,15 +820,14 @@ def test_init_from_config(self): 'argv': ['-c', 'arg2'], 'orig_argv': ['python3', '-W', 'cmdline_warnoption', - '-X', 'cmdline_xoption', + '-X', 'dev', '-c', 'pass', 'arg2'], 'parse_argv': 2, 'xoptions': [ - 'config_xoption1=3', - 'config_xoption2=', - 'config_xoption3', - 'cmdline_xoption', + 'dev=3', + 'utf8', + 'dev', ], 'warnoptions': [ 'cmdline_warnoption', @@ -1046,9 +1045,8 @@ def test_init_sys_add(self): config = { 'faulthandler': 1, 'xoptions': [ - 'config_xoption', - 'cmdline_xoption', - 'sysadd_xoption', + 'dev', + 'utf8', 'faulthandler', ], 'warnoptions': [ @@ -1058,9 +1056,12 @@ def test_init_sys_add(self): ], 'orig_argv': ['python3', '-W', 'ignore:::cmdline_warnoption', - '-X', 'cmdline_xoption'], + '-X', 'utf8'], } - self.check_all_configs("test_init_sys_add", config, api=API_PYTHON) + preconfig = {'utf8_mode': 1} + self.check_all_configs("test_init_sys_add", config, + expected_preconfig=preconfig, + api=API_PYTHON) def test_init_run_main(self): code = ('import _testinternalcapi, json; ' diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-12-14-41-39.bpo-45445._F5cMf.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-12-14-41-39.bpo-45445._F5cMf.rst new file mode 100644 index 0000000000000..d497ae26bd577 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-12-14-41-39.bpo-45445._F5cMf.rst @@ -0,0 +1,2 @@ +Python now fails to initialize if it finds an invalid :option:`-X` option in the +command line. Patch by Pablo Galindo. diff --git a/Programs/_testembed.c b/Programs/_testembed.c index 7628e1a17d9ff..fa418e276114a 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -211,7 +211,7 @@ static int test_pre_initialization_sys_options(void) * relying on the caller to keep the passed in strings alive. */ const wchar_t *static_warnoption = L"once"; - const wchar_t *static_xoption = L"also_not_an_option=2"; + const wchar_t *static_xoption = L"utf8=1"; size_t warnoption_len = wcslen(static_warnoption); size_t xoption_len = wcslen(static_xoption); wchar_t *dynamic_once_warnoption = \ @@ -230,7 +230,7 @@ static int test_pre_initialization_sys_options(void) PySys_AddWarnOption(L"module"); PySys_AddWarnOption(L"default"); _Py_EMBED_PREINIT_CHECK("Checking PySys_AddXOption\n"); - PySys_AddXOption(L"not_an_option=1"); + PySys_AddXOption(L"dev=2"); PySys_AddXOption(dynamic_xoption); /* Delete the dynamic options early */ @@ -548,7 +548,7 @@ static int test_init_from_config(void) L"-W", L"cmdline_warnoption", L"-X", - L"cmdline_xoption", + L"dev", L"-c", L"pass", L"arg2", @@ -556,10 +556,9 @@ static int test_init_from_config(void) config_set_argv(&config, Py_ARRAY_LENGTH(argv), argv); config.parse_argv = 1; - wchar_t* xoptions[3] = { - L"config_xoption1=3", - L"config_xoption2=", - L"config_xoption3", + wchar_t* xoptions[2] = { + L"dev=3", + L"utf8", }; config_set_wide_string_list(&config, &config.xoptions, Py_ARRAY_LENGTH(xoptions), xoptions); @@ -1375,7 +1374,6 @@ static int test_init_read_set(void) static int test_init_sys_add(void) { - PySys_AddXOption(L"sysadd_xoption"); PySys_AddXOption(L"faulthandler"); PySys_AddWarnOption(L"ignore:::sysadd_warnoption"); @@ -1387,14 +1385,14 @@ static int test_init_sys_add(void) L"-W", L"ignore:::cmdline_warnoption", L"-X", - L"cmdline_xoption", + L"utf8", }; config_set_argv(&config, Py_ARRAY_LENGTH(argv), argv); config.parse_argv = 1; PyStatus status; status = PyWideStringList_Append(&config.xoptions, - L"config_xoption"); + L"dev"); if (PyStatus_Exception(status)) { goto fail; } diff --git a/Python/initconfig.c b/Python/initconfig.c index b91d280906cbe..ba6d19db20048 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -2129,6 +2129,49 @@ _PyConfig_InitImportConfig(PyConfig *config) return config_init_import(config, 1); } +// List of known xoptions to validate against the provided ones. Note that all +// options are listed, even if they are only available if a specific macro is +// set, like -X showrefcount which requires a debug build. In this case unknown +// options are silently ignored. +const wchar_t* known_xoptions[] = { + L"faulthandler", + L"showrefcount", + L"tracemalloc", + L"importtime", + L"dev", + L"utf8", + L"pycache_prefix", + L"warn_default_encoding", + L"no_debug_ranges", + L"frozen_modules", + NULL, +}; + +static const wchar_t* +_Py_check_xoptions(const PyWideStringList *xoptions, const wchar_t **names) +{ + for (Py_ssize_t i=0; i < xoptions->length; i++) { + const wchar_t *option = xoptions->items[i]; + size_t len; + wchar_t *sep = wcschr(option, L'='); + if (sep != NULL) { + len = (sep - option); + } + else { + len = wcslen(option); + } + int found = 0; + for (const wchar_t** name = names; *name != NULL; name++) { + if (wcsncmp(option, *name, len) == 0 && (*name)[len] == L'\0') { + found = 1; + } + } + if (found == 0) { + return option; + } + } + return NULL; +} static PyStatus config_read(PyConfig *config, int compute_path_config) @@ -2144,6 +2187,11 @@ config_read(PyConfig *config, int compute_path_config) } /* -X options */ + const wchar_t* option = _Py_check_xoptions(&config->xoptions, known_xoptions); + if (option != NULL) { + return PyStatus_Error("Unknown value for option -X"); + } + if (config_get_xoption(config, L"showrefcount")) { config->show_ref_count = 1; } diff --git a/test_foo.py b/test_foo.py new file mode 100644 index 0000000000000..a27be0fdb47a6 --- /dev/null +++ b/test_foo.py @@ -0,0 +1,3 @@ +def foo(a=3, *, c, d=2): + pass +foo() From webhook-mailer at python.org Wed Oct 13 13:08:49 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 17:08:49 -0000 Subject: [Python-checkins] [3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28929) Message-ID: https://github.com/python/cpython/commit/5d747130933a25a3984f78ae32d2e74a0d2dc5d1 commit: 5d747130933a25a3984f78ae32d2e74a0d2dc5d1 branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-13T19:08:44+02:00 summary: [3.9] bpo-45229: Make tkinter tests discoverable (GH-28637) (GH-28929) (cherry picked from commit f59ed3c310a7ceebf2a56a84ea969a7f75d95b64) Co-authored-by: Serhiy Storchaka files: D Lib/tkinter/test/runtktests.py D Lib/tkinter/test/test_ttk/test_functions.py M Lib/test/test_tk.py M Lib/test/test_ttk_guionly.py M Lib/test/test_ttk_textonly.py M Lib/tkinter/test/test_tkinter/test_colorchooser.py M Lib/tkinter/test/test_tkinter/test_font.py M Lib/tkinter/test/test_tkinter/test_images.py M Lib/tkinter/test/test_tkinter/test_loadtk.py M Lib/tkinter/test/test_tkinter/test_misc.py M Lib/tkinter/test/test_tkinter/test_simpledialog.py M Lib/tkinter/test/test_tkinter/test_text.py M Lib/tkinter/test/test_tkinter/test_variables.py M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/test/test_ttk/test_style.py M Lib/tkinter/test/test_ttk/test_widgets.py M PCbuild/lib.pyproj diff --git a/Lib/test/test_tk.py b/Lib/test/test_tk.py index 48cefd92e1216..d45acbfc6e1a8 100644 --- a/Lib/test/test_tk.py +++ b/Lib/test/test_tk.py @@ -5,11 +5,9 @@ # Skip test if tk cannot be initialized. support.requires('gui') -from tkinter.test import runtktests +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_tkinter') -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_tkinter'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_guionly.py b/Lib/test/test_ttk_guionly.py index 462665db5f3e7..0b57d5a0d6c25 100644 --- a/Lib/test/test_ttk_guionly.py +++ b/Lib/test/test_ttk_guionly.py @@ -10,25 +10,26 @@ import tkinter from _tkinter import TclError from tkinter import ttk -from tkinter.test import runtktests - -root = None -try: - root = tkinter.Tk() - button = ttk.Button(root) - button.destroy() - del button -except TclError as msg: - # assuming ttk is not available - raise unittest.SkipTest("ttk not available: %s" % msg) -finally: - if root is not None: - root.destroy() - del root - -def test_main(): - support.run_unittest( - *runtktests.get_tests(text=False, packages=['test_ttk'])) + + +def setUpModule(): + root = None + try: + root = tkinter.Tk() + button = ttk.Button(root) + button.destroy() + del button + except TclError as msg: + # assuming ttk is not available + raise unittest.SkipTest("ttk not available: %s" % msg) + finally: + if root is not None: + root.destroy() + del root + +def load_tests(loader, tests, pattern): + return loader.discover('tkinter.test.test_ttk') + if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/test/test_ttk_textonly.py b/Lib/test/test_ttk_textonly.py index 7540a4310344c..7a478398a97cf 100644 --- a/Lib/test/test_ttk_textonly.py +++ b/Lib/test/test_ttk_textonly.py @@ -3,11 +3,461 @@ # Skip this test if _tkinter does not exist. support.import_module('_tkinter') -from tkinter.test import runtktests +import unittest +from tkinter import ttk + + +class MockTkApp: + + def splitlist(self, arg): + if isinstance(arg, tuple): + return arg + return arg.split(':') + + def wantobjects(self): + return True + + +class MockTclObj(object): + typename = 'test' + + def __init__(self, val): + self.val = val + + def __str__(self): + return str(self.val) + + +class MockStateSpec(object): + typename = 'StateSpec' + + def __init__(self, *args): + self.val = args + + def __str__(self): + return ' '.join(self.val) + + +class InternalFunctionsTest(unittest.TestCase): + + def test_format_optdict(self): + def check_against(fmt_opts, result): + for i in range(0, len(fmt_opts), 2): + self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) + if result: + self.fail("result still got elements: %s" % result) + + # passing an empty dict should return an empty object (tuple here) + self.assertFalse(ttk._format_optdict({})) + + # check list formatting + check_against( + ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), + {'-fg': 'blue', '-padding': '1 2 3 4'}) + + # check tuple formatting (same as list) + check_against( + ttk._format_optdict({'test': (1, 2, '', 0)}), + {'-test': '1 2 {} 0'}) + + # check untouched values + check_against( + ttk._format_optdict({'test': {'left': 'as is'}}), + {'-test': {'left': 'as is'}}) + + # check script formatting + check_against( + ttk._format_optdict( + {'test': [1, -1, '', '2m', 0], 'test2': 3, + 'test3': '', 'test4': 'abc def', + 'test5': '"abc"', 'test6': '{}', + 'test7': '} -spam {'}, script=True), + {'-test': '{1 -1 {} 2m 0}', '-test2': '3', + '-test3': '{}', '-test4': '{abc def}', + '-test5': '{"abc"}', '-test6': r'\{\}', + '-test7': r'\}\ -spam\ \{'}) + + opts = {'???': True, '?': False} + orig_opts = opts.copy() + # check if giving unicode keys is fine + check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) + # opts should remain unchanged + self.assertEqual(opts, orig_opts) + + # passing values with spaces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one two', 'three')}), + {'-option': '{one two} three'}) + check_against( + ttk._format_optdict( + {'option': ('one\ttwo', 'three')}), + {'-option': '{one\ttwo} three'}) + + # passing empty strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('', 'one')}), + {'-option': '{} one'}) + + # passing values with braces inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('one} {two', 'three')}), + {'-option': r'one\}\ \{two three'}) + + # passing quoted strings inside a tuple/list + check_against( + ttk._format_optdict( + {'option': ('"one"', 'two')}), + {'-option': '{"one"} two'}) + check_against( + ttk._format_optdict( + {'option': ('{one}', 'two')}), + {'-option': r'\{one\} two'}) + + # ignore an option + amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore non-existing options + amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 + self.assertEqual(amount_opts, len(opts) - 1) + + # ignore every option + self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) + + + def test_format_mapdict(self): + opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} + result = ttk._format_mapdict(opts) + self.assertEqual(len(result), len(list(opts.keys())) * 2) + self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) + self.assertEqual(ttk._format_mapdict(opts, script=True), + ('-a', '{{b c} val d otherval {} single}')) + + self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) + + opts = {'?????d?': [('?', 'v?l')]} + result = ttk._format_mapdict(opts) + self.assertEqual(result, ('-?????d?', '? v?l')) + + self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), + ('-opt', '{} value')) + + # empty states + valid = {'opt': [('', '', 'hi')]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) + + # when passing multiple states, they all must be strings + invalid = {'opt': [(1, 2, 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + invalid = {'opt': [([1], '2', 'valid val')]} + self.assertRaises(TypeError, ttk._format_mapdict, invalid) + # but when passing a single state, it can be anything + valid = {'opt': [[1, 'value']]} + self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) + # special attention to single states which evaluate to False + for stateval in (None, 0, False, '', set()): # just some samples + valid = {'opt': [(stateval, 'value')]} + self.assertEqual(ttk._format_mapdict(valid), + ('-opt', '{} value')) + + # values must be iterable + opts = {'a': None} + self.assertRaises(TypeError, ttk._format_mapdict, opts) + + + def test_format_elemcreate(self): + self.assertTrue(ttk._format_elemcreate(None), (None, ())) + + ## Testing type = image + # image type expects at least an image name, so this should raise + # IndexError since it tries to access the index 0 of an empty tuple + self.assertRaises(IndexError, ttk._format_elemcreate, 'image') + + # don't format returned values as a tcl script + # minimum acceptable for image type + self.assertEqual(ttk._format_elemcreate('image', False, 'test'), + ("test ", ())) + # specifying a state spec + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('', 'a')), ("test {} a", ())) + # state spec with multiple states + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b', 'c')), ("test {a b} c", ())) + # state spec and options + self.assertEqual(ttk._format_elemcreate('image', False, 'test', + ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) + # format returned values as a tcl script + # state spec with multiple states and an option with a multivalue + self.assertEqual(ttk._format_elemcreate('image', True, 'test', + ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) + + ## Testing type = vsapi + # vsapi type expects at least a class name and a part_id, so this + # should raise a ValueError since it tries to get two elements from + # an empty tuple + self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') + + # don't format returned values as a tcl script + # minimum acceptable for vsapi + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), + ("a b ", ())) + # now with a state spec with multiple states + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b', 'c')), ("a b {a b} c", ())) + # state spec and option + self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', + ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) + # format returned values as a tcl script + # state spec with a multivalue and an option + self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', + ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) + + # Testing type = from + # from type expects at least a type name + self.assertRaises(IndexError, ttk._format_elemcreate, 'from') + + self.assertEqual(ttk._format_elemcreate('from', False, 'a'), + ('a', ())) + self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), + ('a', ('b', ))) + self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), + ('{a}', 'b')) + + + def test_format_layoutlist(self): + def sample(indent=0, indent_size=2): + return ttk._format_layoutlist( + [('a', {'other': [1, 2, 3], 'children': + [('b', {'children': + [('c', {'children': + [('d', {'nice': 'opt'})], 'something': (1, 2) + })] + })] + })], indent=indent, indent_size=indent_size)[0] + + def sample_expected(indent=0, indent_size=2): + spaces = lambda amount=0: ' ' * (amount + indent) + return ( + "%sa -other {1 2 3} -children {\n" + "%sb -children {\n" + "%sc -something {1 2} -children {\n" + "%sd -nice opt\n" + "%s}\n" + "%s}\n" + "%s}" % (spaces(), spaces(indent_size), + spaces(2 * indent_size), spaces(3 * indent_size), + spaces(2 * indent_size), spaces(indent_size), spaces())) + + # empty layout + self.assertEqual(ttk._format_layoutlist([])[0], '') + + # _format_layoutlist always expects the second item (in every item) + # to act like a dict (except when the value evaluates to False). + self.assertRaises(AttributeError, + ttk._format_layoutlist, [('a', 'b')]) + + smallest = ttk._format_layoutlist([('a', None)], indent=0) + self.assertEqual(smallest, + ttk._format_layoutlist([('a', '')], indent=0)) + self.assertEqual(smallest[0], 'a') + + # testing indentation levels + self.assertEqual(sample(), sample_expected()) + for i in range(4): + self.assertEqual(sample(i), sample_expected(i)) + self.assertEqual(sample(i, i), sample_expected(i, i)) + + # invalid layout format, different kind of exceptions will be + # raised by internal functions + + # plain wrong format + self.assertRaises(ValueError, ttk._format_layoutlist, + ['bad', 'format']) + # will try to use iteritems in the 'bad' string + self.assertRaises(AttributeError, ttk._format_layoutlist, + [('name', 'bad')]) + # bad children formatting + self.assertRaises(ValueError, ttk._format_layoutlist, + [('name', {'children': {'a': None}})]) + + + def test_script_from_settings(self): + # empty options + self.assertFalse(ttk._script_from_settings({'name': + {'configure': None, 'map': None, 'element create': None}})) + + # empty layout + self.assertEqual( + ttk._script_from_settings({'name': {'layout': None}}), + "ttk::style layout name {\nnull\n}") + + configdict = {'???': True, '?': False} + self.assertTrue( + ttk._script_from_settings({'name': {'configure': configdict}})) + + mapdict = {'?????d?': [('?', 'v?l')]} + self.assertTrue( + ttk._script_from_settings({'name': {'map': mapdict}})) + + # invalid image element + self.assertRaises(IndexError, + ttk._script_from_settings, {'name': {'element create': ['image']}}) + + # minimal valid image + self.assertTrue(ttk._script_from_settings({'name': + {'element create': ['image', 'name']}})) + + image = {'thing': {'element create': + ['image', 'name', ('state1', 'state2', 'val')]}} + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} ") + + image['thing']['element create'].append({'opt': 30}) + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt 30") + + image['thing']['element create'][-1]['opt'] = [MockTclObj(3), + MockTclObj('2m')] + self.assertEqual(ttk._script_from_settings(image), + "ttk::style element create thing image {name {state1 state2} val} " + "-opt {3 2m}") + + + def test_tclobj_to_py(self): + self.assertEqual( + ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), + [('a', 'b', 'val')]) + self.assertEqual( + ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), + [1, 2, '3m']) + + + def test_list_from_statespec(self): + def test_it(sspec, value, res_value, states): + self.assertEqual(ttk._list_from_statespec( + (sspec, value)), [states + (res_value, )]) + + states_even = tuple('state%d' % i for i in range(6)) + statespec = MockStateSpec(*states_even) + test_it(statespec, 'val', 'val', states_even) + test_it(statespec, MockTclObj('val'), 'val', states_even) + + states_odd = tuple('state%d' % i for i in range(5)) + statespec = MockStateSpec(*states_odd) + test_it(statespec, 'val', 'val', states_odd) + + test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) + + + def test_list_from_layouttuple(self): + tk = MockTkApp() + + # empty layout tuple + self.assertFalse(ttk._list_from_layouttuple(tk, ())) + + # shortest layout tuple + self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), + [('name', {})]) + + # not so interesting ltuple + sample_ltuple = ('name', '-option', 'value') + self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), + [('name', {'option': 'value'})]) + + # empty children + self.assertEqual(ttk._list_from_layouttuple(tk, + ('something', '-children', ())), + [('something', {'children': []})] + ) + + # more interesting ltuple + ltuple = ( + 'name', '-option', 'niceone', '-children', ( + ('otherone', '-children', ( + ('child', )), '-otheropt', 'othervalue' + ) + ) + ) + self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), + [('name', {'option': 'niceone', 'children': + [('otherone', {'otheropt': 'othervalue', 'children': + [('child', {})] + })] + })] + ) + + # bad tuples + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('name', 'no_minus', 'value')) + self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, + ('something', '-children')) # no children + + + def test_val_or_dict(self): + def func(res, opt=None, val=None): + if opt is None: + return res + if val is None: + return "test val" + return (opt, val) + + tk = MockTkApp() + tk.call = func + + self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), + {'test': '3'}) + self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), + {'test': 3}) + + self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), + 'test val') + + self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), + {'test': 3}) + + + def test_convert_stringval(self): + tests = ( + (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), + (None, 'None') + ) + for orig, expected in tests: + self.assertEqual(ttk._convert_stringval(orig), expected) + + +class TclObjsToPyTest(unittest.TestCase): + + def test_unicode(self): + adict = {'opt': 'v?l??'} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + adict['opt'] = MockTclObj(adict['opt']) + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) + + def test_multivalues(self): + adict = {'opt': [1, 2, 3, 4]} + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) + + adict['opt'] = [1, 'xm', 3] + self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) + + adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') + self.assertEqual(ttk.tclobjs_to_py(adict), + {'opt': [('a', 'b', 'v?l??')]}) + + self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), + {'x': ['y z']}) + + def test_nosplit(self): + self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), + {'text': 'some text'}) -def test_main(): - support.run_unittest( - *runtktests.get_tests(gui=False, packages=['test_ttk'])) if __name__ == '__main__': - test_main() + unittest.main() diff --git a/Lib/tkinter/test/runtktests.py b/Lib/tkinter/test/runtktests.py deleted file mode 100644 index 33dc54a1375bf..0000000000000 --- a/Lib/tkinter/test/runtktests.py +++ /dev/null @@ -1,69 +0,0 @@ -""" -Use this module to get and run all tk tests. - -tkinter tests should live in a package inside the directory where this file -lives, like test_tkinter. -Extensions also should live in packages following the same rule as above. -""" - -import os -import importlib -import test.support - -this_dir_path = os.path.abspath(os.path.dirname(__file__)) - -def is_package(path): - for name in os.listdir(path): - if name in ('__init__.py', '__init__.pyc'): - return True - return False - -def get_tests_modules(basepath=this_dir_path, gui=True, packages=None): - """This will import and yield modules whose names start with test_ - and are inside packages found in the path starting at basepath. - - If packages is specified it should contain package names that - want their tests collected. - """ - py_ext = '.py' - - for dirpath, dirnames, filenames in os.walk(basepath): - for dirname in list(dirnames): - if dirname[0] == '.': - dirnames.remove(dirname) - - if is_package(dirpath) and filenames: - pkg_name = dirpath[len(basepath) + len(os.sep):].replace('/', '.') - if packages and pkg_name not in packages: - continue - - filenames = filter( - lambda x: x.startswith('test_') and x.endswith(py_ext), - filenames) - - for name in filenames: - try: - yield importlib.import_module( - ".%s.%s" % (pkg_name, name[:-len(py_ext)]), - "tkinter.test") - except test.support.ResourceDenied: - if gui: - raise - -def get_tests(text=True, gui=True, packages=None): - """Yield all the tests in the modules found by get_tests_modules. - - If nogui is True, only tests that do not require a GUI will be - returned.""" - attrs = [] - if text: - attrs.append('tests_nogui') - if gui: - attrs.append('tests_gui') - for module in get_tests_modules(gui=gui, packages=packages): - for attr in attrs: - for test in getattr(module, attr, ()): - yield test - -if __name__ == "__main__": - test.support.run_unittest(*get_tests()) diff --git a/Lib/tkinter/test/test_tkinter/test_colorchooser.py b/Lib/tkinter/test/test_tkinter/test_colorchooser.py index 4798bc9c26fc0..ce6644af44bbc 100644 --- a/Lib/tkinter/test/test_tkinter/test_colorchooser.py +++ b/Lib/tkinter/test/test_tkinter/test_colorchooser.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractTkTest from tkinter import colorchooser @@ -37,7 +37,5 @@ def test_fixresult(self): ((74, 60, 140), '#4a3c8c')) -tests_gui = (ChooserTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_font.py b/Lib/tkinter/test/test_tkinter/test_font.py index 0ea7924f5d5c9..9573d4c8a05c9 100644 --- a/Lib/tkinter/test/test_tkinter/test_font.py +++ b/Lib/tkinter/test/test_tkinter/test_font.py @@ -1,7 +1,7 @@ import unittest import tkinter from tkinter import font -from test.support import requires, run_unittest, gc_collect, ALWAYS_EQ +from test.support import requires, gc_collect, ALWAYS_EQ from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -139,7 +139,5 @@ def test_names(self): self.assertRaises(RuntimeError, font.names) -tests_gui = (FontTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_images.py b/Lib/tkinter/test/test_tkinter/test_images.py index c798966e0b69f..9e57f02a69600 100644 --- a/Lib/tkinter/test/test_tkinter/test_images.py +++ b/Lib/tkinter/test/test_tkinter/test_images.py @@ -375,7 +375,5 @@ def test_transparency(self): self.assertEqual(image.transparency_get(4, 6), False) -tests_gui = (MiscTest, DefaultRootTest, BitmapImageTest, PhotoImageTest,) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_loadtk.py b/Lib/tkinter/test/test_tkinter/test_loadtk.py index bab7bcd37ce31..d97ae6d456f7b 100644 --- a/Lib/tkinter/test/test_tkinter/test_loadtk.py +++ b/Lib/tkinter/test/test_tkinter/test_loadtk.py @@ -40,7 +40,6 @@ def testLoadTkFailure(self): self.assertRaises(TclError, tcl.winfo_geometry) self.assertRaises(TclError, tcl.loadtk) -tests_gui = (TkLoadTest, ) if __name__ == "__main__": - test_support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index f6e5b4db1ae1f..554ae74d6b415 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -339,7 +339,5 @@ def test_mainloop(self): self.assertRaises(RuntimeError, tkinter.mainloop) -tests_gui = (MiscTest, DefaultRootTest) - if __name__ == "__main__": - support.run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_simpledialog.py b/Lib/tkinter/test/test_tkinter/test_simpledialog.py index 911917258806d..6d77ea6331870 100644 --- a/Lib/tkinter/test/test_tkinter/test_simpledialog.py +++ b/Lib/tkinter/test/test_tkinter/test_simpledialog.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest, swap_attr +from test.support import requires, swap_attr from tkinter.test.support import AbstractDefaultRootTest from tkinter.simpledialog import Dialog, askinteger @@ -19,7 +19,5 @@ def test_askinteger(self): self.assertRaises(RuntimeError, askinteger, "Go To Line", "Line number") -tests_gui = (DefaultRootTest,) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_text.py b/Lib/tkinter/test/test_tkinter/test_text.py index 13b7c56a3978d..482f150df559f 100644 --- a/Lib/tkinter/test/test_tkinter/test_text.py +++ b/Lib/tkinter/test/test_tkinter/test_text.py @@ -1,6 +1,6 @@ import unittest import tkinter -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -41,7 +41,5 @@ def test_search(self): self.assertEqual(text.search('test', '1.0', 'end'), '1.3') -tests_gui = (TextTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_tkinter/test_variables.py b/Lib/tkinter/test/test_tkinter/test_variables.py index 0be5282a3a3b3..427e168454362 100644 --- a/Lib/tkinter/test/test_tkinter/test_variables.py +++ b/Lib/tkinter/test/test_tkinter/test_variables.py @@ -338,10 +338,5 @@ def test_variable(self): self.assertRaises(RuntimeError, Variable) -tests_gui = (TestVariable, TestStringVar, TestIntVar, - TestDoubleVar, TestBooleanVar, DefaultRootTest) - - if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index e6b3eccf7afb8..438d21d0b3733 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -2,7 +2,7 @@ import unittest import tkinter from tkinter import ttk -from test.support import requires, run_unittest, gc_collect +from test.support import requires, gc_collect from tkinter.test.support import AbstractTkTest, AbstractDefaultRootTest requires('gui') @@ -308,7 +308,5 @@ def test_labeledscale(self): self._test_widget(ttk.LabeledScale) -tests_gui = (LabeledScaleTest, OptionMenuTest, DefaultRootTest) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_functions.py b/Lib/tkinter/test/test_ttk/test_functions.py deleted file mode 100644 index 5c23d6fecf8b4..0000000000000 --- a/Lib/tkinter/test/test_ttk/test_functions.py +++ /dev/null @@ -1,460 +0,0 @@ -# -*- encoding: utf-8 -*- -import unittest -from tkinter import ttk - -class MockTkApp: - - def splitlist(self, arg): - if isinstance(arg, tuple): - return arg - return arg.split(':') - - def wantobjects(self): - return True - - -class MockTclObj(object): - typename = 'test' - - def __init__(self, val): - self.val = val - - def __str__(self): - return str(self.val) - - -class MockStateSpec(object): - typename = 'StateSpec' - - def __init__(self, *args): - self.val = args - - def __str__(self): - return ' '.join(self.val) - - -class InternalFunctionsTest(unittest.TestCase): - - def test_format_optdict(self): - def check_against(fmt_opts, result): - for i in range(0, len(fmt_opts), 2): - self.assertEqual(result.pop(fmt_opts[i]), fmt_opts[i + 1]) - if result: - self.fail("result still got elements: %s" % result) - - # passing an empty dict should return an empty object (tuple here) - self.assertFalse(ttk._format_optdict({})) - - # check list formatting - check_against( - ttk._format_optdict({'fg': 'blue', 'padding': [1, 2, 3, 4]}), - {'-fg': 'blue', '-padding': '1 2 3 4'}) - - # check tuple formatting (same as list) - check_against( - ttk._format_optdict({'test': (1, 2, '', 0)}), - {'-test': '1 2 {} 0'}) - - # check untouched values - check_against( - ttk._format_optdict({'test': {'left': 'as is'}}), - {'-test': {'left': 'as is'}}) - - # check script formatting - check_against( - ttk._format_optdict( - {'test': [1, -1, '', '2m', 0], 'test2': 3, - 'test3': '', 'test4': 'abc def', - 'test5': '"abc"', 'test6': '{}', - 'test7': '} -spam {'}, script=True), - {'-test': '{1 -1 {} 2m 0}', '-test2': '3', - '-test3': '{}', '-test4': '{abc def}', - '-test5': '{"abc"}', '-test6': r'\{\}', - '-test7': r'\}\ -spam\ \{'}) - - opts = {'???': True, '?': False} - orig_opts = opts.copy() - # check if giving unicode keys is fine - check_against(ttk._format_optdict(opts), {'-???': True, '-?': False}) - # opts should remain unchanged - self.assertEqual(opts, orig_opts) - - # passing values with spaces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one two', 'three')}), - {'-option': '{one two} three'}) - check_against( - ttk._format_optdict( - {'option': ('one\ttwo', 'three')}), - {'-option': '{one\ttwo} three'}) - - # passing empty strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('', 'one')}), - {'-option': '{} one'}) - - # passing values with braces inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('one} {two', 'three')}), - {'-option': r'one\}\ \{two three'}) - - # passing quoted strings inside a tuple/list - check_against( - ttk._format_optdict( - {'option': ('"one"', 'two')}), - {'-option': '{"one"} two'}) - check_against( - ttk._format_optdict( - {'option': ('{one}', 'two')}), - {'-option': r'\{one\} two'}) - - # ignore an option - amount_opts = len(ttk._format_optdict(opts, ignore=('?'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore non-existing options - amount_opts = len(ttk._format_optdict(opts, ignore=('?', 'b'))) / 2 - self.assertEqual(amount_opts, len(opts) - 1) - - # ignore every option - self.assertFalse(ttk._format_optdict(opts, ignore=list(opts.keys()))) - - - def test_format_mapdict(self): - opts = {'a': [('b', 'c', 'val'), ('d', 'otherval'), ('', 'single')]} - result = ttk._format_mapdict(opts) - self.assertEqual(len(result), len(list(opts.keys())) * 2) - self.assertEqual(result, ('-a', '{b c} val d otherval {} single')) - self.assertEqual(ttk._format_mapdict(opts, script=True), - ('-a', '{{b c} val d otherval {} single}')) - - self.assertEqual(ttk._format_mapdict({2: []}), ('-2', '')) - - opts = {'?????d?': [('?', 'v?l')]} - result = ttk._format_mapdict(opts) - self.assertEqual(result, ('-?????d?', '? v?l')) - - self.assertEqual(ttk._format_mapdict({'opt': [('value',)]}), - ('-opt', '{} value')) - - # empty states - valid = {'opt': [('', '', 'hi')]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '{ } hi')) - - # when passing multiple states, they all must be strings - invalid = {'opt': [(1, 2, 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - invalid = {'opt': [([1], '2', 'valid val')]} - self.assertRaises(TypeError, ttk._format_mapdict, invalid) - # but when passing a single state, it can be anything - valid = {'opt': [[1, 'value']]} - self.assertEqual(ttk._format_mapdict(valid), ('-opt', '1 value')) - # special attention to single states which evaluate to False - for stateval in (None, 0, False, '', set()): # just some samples - valid = {'opt': [(stateval, 'value')]} - self.assertEqual(ttk._format_mapdict(valid), - ('-opt', '{} value')) - - # values must be iterable - opts = {'a': None} - self.assertRaises(TypeError, ttk._format_mapdict, opts) - - - def test_format_elemcreate(self): - self.assertTrue(ttk._format_elemcreate(None), (None, ())) - - ## Testing type = image - # image type expects at least an image name, so this should raise - # IndexError since it tries to access the index 0 of an empty tuple - self.assertRaises(IndexError, ttk._format_elemcreate, 'image') - - # don't format returned values as a tcl script - # minimum acceptable for image type - self.assertEqual(ttk._format_elemcreate('image', False, 'test'), - ("test ", ())) - # specifying a state spec - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('', 'a')), ("test {} a", ())) - # state spec with multiple states - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b', 'c')), ("test {a b} c", ())) - # state spec and options - self.assertEqual(ttk._format_elemcreate('image', False, 'test', - ('a', 'b'), a='x'), ("test a b", ("-a", "x"))) - # format returned values as a tcl script - # state spec with multiple states and an option with a multivalue - self.assertEqual(ttk._format_elemcreate('image', True, 'test', - ('a', 'b', 'c', 'd'), x=[2, 3]), ("{test {a b c} d}", "-x {2 3}")) - - ## Testing type = vsapi - # vsapi type expects at least a class name and a part_id, so this - # should raise a ValueError since it tries to get two elements from - # an empty tuple - self.assertRaises(ValueError, ttk._format_elemcreate, 'vsapi') - - # don't format returned values as a tcl script - # minimum acceptable for vsapi - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b'), - ("a b ", ())) - # now with a state spec with multiple states - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b', 'c')), ("a b {a b} c", ())) - # state spec and option - self.assertEqual(ttk._format_elemcreate('vsapi', False, 'a', 'b', - ('a', 'b'), opt='x'), ("a b a b", ("-opt", "x"))) - # format returned values as a tcl script - # state spec with a multivalue and an option - self.assertEqual(ttk._format_elemcreate('vsapi', True, 'a', 'b', - ('a', 'b', [1, 2]), opt='x'), ("{a b {a b} {1 2}}", "-opt x")) - - # Testing type = from - # from type expects at least a type name - self.assertRaises(IndexError, ttk._format_elemcreate, 'from') - - self.assertEqual(ttk._format_elemcreate('from', False, 'a'), - ('a', ())) - self.assertEqual(ttk._format_elemcreate('from', False, 'a', 'b'), - ('a', ('b', ))) - self.assertEqual(ttk._format_elemcreate('from', True, 'a', 'b'), - ('{a}', 'b')) - - - def test_format_layoutlist(self): - def sample(indent=0, indent_size=2): - return ttk._format_layoutlist( - [('a', {'other': [1, 2, 3], 'children': - [('b', {'children': - [('c', {'children': - [('d', {'nice': 'opt'})], 'something': (1, 2) - })] - })] - })], indent=indent, indent_size=indent_size)[0] - - def sample_expected(indent=0, indent_size=2): - spaces = lambda amount=0: ' ' * (amount + indent) - return ( - "%sa -other {1 2 3} -children {\n" - "%sb -children {\n" - "%sc -something {1 2} -children {\n" - "%sd -nice opt\n" - "%s}\n" - "%s}\n" - "%s}" % (spaces(), spaces(indent_size), - spaces(2 * indent_size), spaces(3 * indent_size), - spaces(2 * indent_size), spaces(indent_size), spaces())) - - # empty layout - self.assertEqual(ttk._format_layoutlist([])[0], '') - - # _format_layoutlist always expects the second item (in every item) - # to act like a dict (except when the value evaluates to False). - self.assertRaises(AttributeError, - ttk._format_layoutlist, [('a', 'b')]) - - smallest = ttk._format_layoutlist([('a', None)], indent=0) - self.assertEqual(smallest, - ttk._format_layoutlist([('a', '')], indent=0)) - self.assertEqual(smallest[0], 'a') - - # testing indentation levels - self.assertEqual(sample(), sample_expected()) - for i in range(4): - self.assertEqual(sample(i), sample_expected(i)) - self.assertEqual(sample(i, i), sample_expected(i, i)) - - # invalid layout format, different kind of exceptions will be - # raised by internal functions - - # plain wrong format - self.assertRaises(ValueError, ttk._format_layoutlist, - ['bad', 'format']) - # will try to use iteritems in the 'bad' string - self.assertRaises(AttributeError, ttk._format_layoutlist, - [('name', 'bad')]) - # bad children formatting - self.assertRaises(ValueError, ttk._format_layoutlist, - [('name', {'children': {'a': None}})]) - - - def test_script_from_settings(self): - # empty options - self.assertFalse(ttk._script_from_settings({'name': - {'configure': None, 'map': None, 'element create': None}})) - - # empty layout - self.assertEqual( - ttk._script_from_settings({'name': {'layout': None}}), - "ttk::style layout name {\nnull\n}") - - configdict = {'???': True, '?': False} - self.assertTrue( - ttk._script_from_settings({'name': {'configure': configdict}})) - - mapdict = {'?????d?': [('?', 'v?l')]} - self.assertTrue( - ttk._script_from_settings({'name': {'map': mapdict}})) - - # invalid image element - self.assertRaises(IndexError, - ttk._script_from_settings, {'name': {'element create': ['image']}}) - - # minimal valid image - self.assertTrue(ttk._script_from_settings({'name': - {'element create': ['image', 'name']}})) - - image = {'thing': {'element create': - ['image', 'name', ('state1', 'state2', 'val')]}} - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} ") - - image['thing']['element create'].append({'opt': 30}) - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt 30") - - image['thing']['element create'][-1]['opt'] = [MockTclObj(3), - MockTclObj('2m')] - self.assertEqual(ttk._script_from_settings(image), - "ttk::style element create thing image {name {state1 state2} val} " - "-opt {3 2m}") - - - def test_tclobj_to_py(self): - self.assertEqual( - ttk._tclobj_to_py((MockStateSpec('a', 'b'), 'val')), - [('a', 'b', 'val')]) - self.assertEqual( - ttk._tclobj_to_py([MockTclObj('1'), 2, MockTclObj('3m')]), - [1, 2, '3m']) - - - def test_list_from_statespec(self): - def test_it(sspec, value, res_value, states): - self.assertEqual(ttk._list_from_statespec( - (sspec, value)), [states + (res_value, )]) - - states_even = tuple('state%d' % i for i in range(6)) - statespec = MockStateSpec(*states_even) - test_it(statespec, 'val', 'val', states_even) - test_it(statespec, MockTclObj('val'), 'val', states_even) - - states_odd = tuple('state%d' % i for i in range(5)) - statespec = MockStateSpec(*states_odd) - test_it(statespec, 'val', 'val', states_odd) - - test_it(('a', 'b', 'c'), MockTclObj('val'), 'val', ('a', 'b', 'c')) - - - def test_list_from_layouttuple(self): - tk = MockTkApp() - - # empty layout tuple - self.assertFalse(ttk._list_from_layouttuple(tk, ())) - - # shortest layout tuple - self.assertEqual(ttk._list_from_layouttuple(tk, ('name', )), - [('name', {})]) - - # not so interesting ltuple - sample_ltuple = ('name', '-option', 'value') - self.assertEqual(ttk._list_from_layouttuple(tk, sample_ltuple), - [('name', {'option': 'value'})]) - - # empty children - self.assertEqual(ttk._list_from_layouttuple(tk, - ('something', '-children', ())), - [('something', {'children': []})] - ) - - # more interesting ltuple - ltuple = ( - 'name', '-option', 'niceone', '-children', ( - ('otherone', '-children', ( - ('child', )), '-otheropt', 'othervalue' - ) - ) - ) - self.assertEqual(ttk._list_from_layouttuple(tk, ltuple), - [('name', {'option': 'niceone', 'children': - [('otherone', {'otheropt': 'othervalue', 'children': - [('child', {})] - })] - })] - ) - - # bad tuples - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('name', 'no_minus', 'value')) - self.assertRaises(ValueError, ttk._list_from_layouttuple, tk, - ('something', '-children')) # no children - - - def test_val_or_dict(self): - def func(res, opt=None, val=None): - if opt is None: - return res - if val is None: - return "test val" - return (opt, val) - - tk = MockTkApp() - tk.call = func - - self.assertEqual(ttk._val_or_dict(tk, {}, '-test:3'), - {'test': '3'}) - self.assertEqual(ttk._val_or_dict(tk, {}, ('-test', 3)), - {'test': 3}) - - self.assertEqual(ttk._val_or_dict(tk, {'test': None}, 'x:y'), - 'test val') - - self.assertEqual(ttk._val_or_dict(tk, {'test': 3}, 'x:y'), - {'test': 3}) - - - def test_convert_stringval(self): - tests = ( - (0, 0), ('09', 9), ('a', 'a'), ('??', '??'), ([], '[]'), - (None, 'None') - ) - for orig, expected in tests: - self.assertEqual(ttk._convert_stringval(orig), expected) - - -class TclObjsToPyTest(unittest.TestCase): - - def test_unicode(self): - adict = {'opt': 'v?l??'} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - adict['opt'] = MockTclObj(adict['opt']) - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': 'v?l??'}) - - def test_multivalues(self): - adict = {'opt': [1, 2, 3, 4]} - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 2, 3, 4]}) - - adict['opt'] = [1, 'xm', 3] - self.assertEqual(ttk.tclobjs_to_py(adict), {'opt': [1, 'xm', 3]}) - - adict['opt'] = (MockStateSpec('a', 'b'), 'v?l??') - self.assertEqual(ttk.tclobjs_to_py(adict), - {'opt': [('a', 'b', 'v?l??')]}) - - self.assertEqual(ttk.tclobjs_to_py({'x': ['y z']}), - {'x': ['y z']}) - - def test_nosplit(self): - self.assertEqual(ttk.tclobjs_to_py({'text': 'some text'}), - {'text': 'some text'}) - -tests_nogui = (InternalFunctionsTest, TclObjsToPyTest) - -if __name__ == "__main__": - from test.support import run_unittest - run_unittest(*tests_nogui) diff --git a/Lib/tkinter/test/test_ttk/test_style.py b/Lib/tkinter/test/test_ttk/test_style.py index 38d70d7a89077..a33c24ac55bee 100644 --- a/Lib/tkinter/test/test_ttk/test_style.py +++ b/Lib/tkinter/test/test_ttk/test_style.py @@ -3,7 +3,7 @@ import tkinter from tkinter import ttk from test import support -from test.support import requires, run_unittest +from test.support import requires from tkinter.test.support import AbstractTkTest requires('gui') @@ -175,7 +175,5 @@ def test_map_custom_copy(self): self.assertEqual(style.map(newname, key), value) -tests_gui = (StyleTest, ) - if __name__ == "__main__": - run_unittest(*tests_gui) + unittest.main() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 082da5d0c1a00..a2a4de2e5cc41 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -4,7 +4,7 @@ from test.support import requires, gc_collect import sys -from tkinter.test.test_ttk.test_functions import MockTclObj +from test.test_ttk_textonly import MockTclObj from tkinter.test.support import (AbstractTkTest, tcl_version, get_tk_patchlevel, simulate_mouse_click, AbstractDefaultRootTest) from tkinter.test.widget_tests import (add_standard_options, noconv, diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index f0c51edb9d1ca..0fc2aa042d758 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -1430,7 +1430,6 @@ - @@ -1442,7 +1441,6 @@ - From webhook-mailer at python.org Wed Oct 13 13:12:27 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 17:12:27 -0000 Subject: [Python-checkins] [3.10] bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) (GH-28930) Message-ID: https://github.com/python/cpython/commit/f8473f6f7603f8cccccc3307d4cb853587be41b3 commit: f8473f6f7603f8cccccc3307d4cb853587be41b3 branch: 3.10 author: ?ukasz Langa committer: ambv date: 2021-10-13T19:12:22+02:00 summary: [3.10] bpo-45239: Fix parsedate_tz when time has more than 2 dots in it (GH-28452) (GH-28930) Co-authored-by: ?ukasz Langa (cherry picked from commit b9e687618d3489944f29adbd2be50b46940c9e70) Co-authored-by: Ben Hoyt files: A Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst M Lib/email/_parseaddr.py M Lib/test/test_email/test_email.py diff --git a/Lib/email/_parseaddr.py b/Lib/email/_parseaddr.py index 977fedf67b159..ba5ad5a36d06b 100644 --- a/Lib/email/_parseaddr.py +++ b/Lib/email/_parseaddr.py @@ -128,6 +128,8 @@ def _parsedate_tz(data): tss = 0 elif len(tm) == 3: [thh, tmm, tss] = tm + else: + return None else: return None try: diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py index 4001f716471dc..54ffcdc544e8b 100644 --- a/Lib/test/test_email/test_email.py +++ b/Lib/test/test_email/test_email.py @@ -3009,6 +3009,7 @@ def test_parsedate_returns_None_for_invalid_strings(self): self.assertIsNone(utils.parsedate_tz('0')) self.assertIsNone(utils.parsedate('A Complete Waste of Time')) self.assertIsNone(utils.parsedate_tz('A Complete Waste of Time')) + self.assertIsNone(utils.parsedate_tz('Wed, 3 Apr 2002 12.34.56.78+0800')) # Not a part of the spec but, but this has historically worked: self.assertIsNone(utils.parsedate(None)) self.assertIsNone(utils.parsedate_tz(None)) diff --git a/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst new file mode 100644 index 0000000000000..9e5ec561c362a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-13-17-52-48.bpo-45239.7li1_0.rst @@ -0,0 +1,3 @@ +Fixed :func:`email.utils.parsedate_tz` crashing with +:exc:`UnboundLocalError` on certain invalid input instead of returning +``None``. Patch by Ben Hoyt. From webhook-mailer at python.org Wed Oct 13 13:15:48 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 17:15:48 -0000 Subject: [Python-checkins] bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) (GH-28931) Message-ID: https://github.com/python/cpython/commit/fd2be6da2ffec2301a58aefa92f17f0c567fbc55 commit: fd2be6da2ffec2301a58aefa92f17f0c567fbc55 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T19:15:43+02:00 summary: bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) (GH-28931) (cherry picked from commit 6fafc25aea8689048314b5bf7a9bb986bb1ce238) Co-authored-by: andrei kulakov files: A Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index ce4635adcbfad..f1f6f3227bd17 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -526,12 +526,13 @@ def _format_action(self, action): parts = [action_header] # if there was help for the action, add lines of help text - if action.help: + if action.help and action.help.strip(): help_text = self._expand_help(action) - help_lines = self._split_lines(help_text, help_width) - parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) - for line in help_lines[1:]: - parts.append('%*s%s\n' % (help_position, '', line)) + if help_text: + help_lines = self._split_lines(help_text, help_width) + parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) + for line in help_lines[1:]: + parts.append('%*s%s\n' % (help_position, '', line)) # or add a newline if the description doesn't end with one elif not action_header.endswith('\n'): diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 0927281f69c92..6680feb016a4d 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2145,6 +2145,42 @@ def test_help_non_breaking_spaces(self): wrap\N{NO-BREAK SPACE}at non-breaking spaces ''')) + def test_help_blank(self): + # Issue 24444 + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', + help=' ') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] foo + + main description + + positional arguments: + foo + + options: + -h, --help show this help message and exit + ''')) + + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', choices=[], + help='%(choices)s') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] {} + + main description + + positional arguments: + {} + + options: + -h, --help show this help message and exit + ''')) + def test_help_alternate_prefix_chars(self): parser = self._get_parser(prefix_chars='+:/') self.assertEqual(parser.format_usage(), diff --git a/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst new file mode 100644 index 0000000000000..efcacb8f0eb59 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst @@ -0,0 +1,2 @@ +Fixed an error raised in :mod:`argparse` help display when help for an +option is set to 1+ blank spaces or when *choices* arg is an empty container. From webhook-mailer at python.org Wed Oct 13 13:26:00 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 17:26:00 -0000 Subject: [Python-checkins] bpo-45434: pyport.h no longer includes (GH-28914) Message-ID: https://github.com/python/cpython/commit/aac29af6785712019d34f1a7f15b3c408a4f68ae commit: aac29af6785712019d34f1a7f15b3c408a4f68ae branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T19:25:53+02:00 summary: bpo-45434: pyport.h no longer includes (GH-28914) Include explicitly in C files. Python.h includes . files: M Doc/whatsnew/3.11.rst M Include/Python.h M Include/internal/pycore_fileutils.h M Include/pyport.h M Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst M Modules/_ctypes/_ctypes_test.c M Modules/_gdbmmodule.c M Modules/_lzmamodule.c M Modules/_pickle.c M Modules/_tracemalloc.c M Modules/faulthandler.c M Modules/getpath.c M Modules/nismodule.c M Modules/ossaudiodev.c M Modules/posixmodule.c M Modules/readline.c M Objects/floatobject.c M Objects/longobject.c M Objects/obmalloc.c M PC/WinMain.c M Programs/_freeze_module.c M Programs/_testembed.c M Python/dtoa.c M Python/errors.c M Python/fileutils.c M Python/initconfig.c M Python/preconfig.c M Python/pylifecycle.c M Python/pystrhex.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 21a0e1ab9c693..d18711d0b8df2 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -556,6 +556,9 @@ Porting to Python 3.11 * The ```` header file no longer includes ````. C extensions using ```` must now include it explicitly. + The system ```` header provides functions like: + ``malloc()``/``free()``, ``getenv()``, ``strtol()``, ``abs()``, ``strtol()``, + ``exit()`` and ``abort()``. (Contributed by Victor Stinner in :issue:`45434`.) Deprecated diff --git a/Include/Python.h b/Include/Python.h index 8c84c90f9a57e..4f62103e30276 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -33,7 +33,8 @@ # include #endif -#include +#include // assert() +#include // wchar_t #include "pyport.h" #include "pymacro.h" diff --git a/Include/internal/pycore_fileutils.h b/Include/internal/pycore_fileutils.h index 2316a978294e2..3464477bce575 100644 --- a/Include/internal/pycore_fileutils.h +++ b/Include/internal/pycore_fileutils.h @@ -88,6 +88,9 @@ extern int _Py_add_relfile(wchar_t *dirname, // ... // _Py_END_SUPPRESS_IPH #if defined _MSC_VER && _MSC_VER >= 1900 + +# include // _set_thread_local_invalid_parameter_handler() + extern _invalid_parameter_handler _Py_silent_invalid_parameter_handler; # define _Py_BEGIN_SUPPRESS_IPH \ { _invalid_parameter_handler _Py_old_handler = \ diff --git a/Include/pyport.h b/Include/pyport.h index 6e4e98020cb1f..0bec2a9b38f7c 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -205,8 +205,6 @@ typedef Py_ssize_t Py_ssize_clean_t; * see https://bugs.python.org/issue28126 */ #define Py_MEMCPY memcpy -#include - #ifdef HAVE_IEEEFP_H #include /* needed for 'finite' declaration on some platforms */ #endif diff --git a/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst index c04dda532557d..95c5f0d1150c8 100644 --- a/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst +++ b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst @@ -1,3 +1,6 @@ The ```` header file no longer includes ````. C -extensions using ```` must now include it explicitly. Patch by -Victor Stinner. +extensions using ```` must now include it explicitly. +The system ```` header provides functions like: +``malloc()``/``free()``, ``getenv()``, ``strtol()``, ``abs()``, ``strtol()``, +``exit()`` and ``abort()``. +Patch by Victor Stinner. diff --git a/Modules/_ctypes/_ctypes_test.c b/Modules/_ctypes/_ctypes_test.c index a33d15de9c0d4..770c96c60d1f7 100644 --- a/Modules/_ctypes/_ctypes_test.c +++ b/Modules/_ctypes/_ctypes_test.c @@ -4,6 +4,8 @@ #include #endif +#include // qsort() + #define EXPORT(x) Py_EXPORTED_SYMBOL x /* some functions handy for testing */ diff --git a/Modules/_gdbmmodule.c b/Modules/_gdbmmodule.c index efbf331ca302d..445500c7ee4c1 100644 --- a/Modules/_gdbmmodule.c +++ b/Modules/_gdbmmodule.c @@ -5,11 +5,12 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "gdbm.h" -#include -#include #include -#include "gdbm.h" +#include // free() +#include +#include #if defined(WIN32) && !defined(__CYGWIN__) #include "gdbmerrno.h" diff --git a/Modules/_lzmamodule.c b/Modules/_lzmamodule.c index a7156ec7ddf04..e50f55b75c5dc 100644 --- a/Modules/_lzmamodule.c +++ b/Modules/_lzmamodule.c @@ -10,6 +10,7 @@ #include "Python.h" #include "structmember.h" // PyMemberDef +#include // free() #include #include diff --git a/Modules/_pickle.c b/Modules/_pickle.c index e693b506a1e91..b5131696981dc 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -12,6 +12,8 @@ #include "pycore_moduleobject.h" // _PyModule_GetState() #include "structmember.h" // PyMemberDef +#include // strtol() + PyDoc_STRVAR(pickle_module_doc, "Optimized C implementation for the Python pickle module."); diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 9ba0ebbbaab5c..09d273ad280bb 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -5,7 +5,10 @@ #include "pycore_hashtable.h" #include +#include // malloc() + #include "clinic/_tracemalloc.c.h" + /*[clinic input] module _tracemalloc [clinic start generated code]*/ diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index 868b4f4f42de6..cb2e2588e19b2 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -3,10 +3,13 @@ #include "pycore_pyerrors.h" // _Py_DumpExtensionModules #include "pycore_pystate.h" // _PyThreadState_GET() #include "pycore_traceback.h" // _Py_DumpTracebackThreads -#include + +#include "frameobject.h" + #include -#include #include +#include +#include // abort() #if defined(HAVE_PTHREAD_SIGMASK) && !defined(HAVE_BROKEN_PTHREAD_SIGMASK) # include #endif diff --git a/Modules/getpath.c b/Modules/getpath.c index 22e5ef24b5d57..1405023b39b58 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -6,8 +6,9 @@ #include "pycore_pathconfig.h" #include "osdefs.h" // DELIM -#include +#include // getenv() #include +#include #ifdef __APPLE__ # include diff --git a/Modules/nismodule.c b/Modules/nismodule.c index 6655451ebd2fb..cdda1a6a2fb7d 100644 --- a/Modules/nismodule.c +++ b/Modules/nismodule.c @@ -12,6 +12,7 @@ #include "Python.h" +#include // free() #include #include #include diff --git a/Modules/ossaudiodev.c b/Modules/ossaudiodev.c index 4f2d9cb8b7c9c..4bab9a58eb104 100644 --- a/Modules/ossaudiodev.c +++ b/Modules/ossaudiodev.c @@ -21,6 +21,7 @@ #include "Python.h" #include "structmember.h" // PyMemberDef +#include // getenv() #ifdef HAVE_FCNTL_H #include #else diff --git a/Modules/posixmodule.c b/Modules/posixmodule.c index 9c174ee6b18df..667a3339f5ba8 100644 --- a/Modules/posixmodule.c +++ b/Modules/posixmodule.c @@ -46,7 +46,8 @@ # undef HAVE_FACCESSAT #endif -#include /* needed for ctermid() */ +#include // ctermid() +#include // system() /* * A number of APIs are available on macOS from a certain macOS version. diff --git a/Modules/readline.c b/Modules/readline.c index c79d22f85f84e..1b616fc4f3b4e 100644 --- a/Modules/readline.c +++ b/Modules/readline.c @@ -6,9 +6,11 @@ /* Standard definitions */ #include "Python.h" -#include -#include + #include +#include +#include +#include // free() #include #if defined(HAVE_SETLOCALE) diff --git a/Objects/floatobject.c b/Objects/floatobject.c index d25d97f4cbf61..5a8113eca8bd0 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -13,6 +13,7 @@ #include #include +#include // strtol() /*[clinic input] class float "PyObject *" "&PyFloat_Type" diff --git a/Objects/longobject.c b/Objects/longobject.c index 33fea6491b73d..111ef4566cda8 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -10,9 +10,10 @@ #include "pycore_pystate.h" // _Py_IsMainInterpreter() #include "longintrepr.h" -#include #include +#include #include +#include // abs() #include "clinic/longobject.c.h" /*[clinic input] diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 2d6fedd6cf0a1..d8d6f6dea0d53 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -2,6 +2,7 @@ #include "pycore_pymem.h" // _PyTraceMalloc_Config #include +#include // malloc() /* Defined in tracemalloc.c */ diff --git a/PC/WinMain.c b/PC/WinMain.c index e439bed9ed113..07e21ce3e9f2d 100644 --- a/PC/WinMain.c +++ b/PC/WinMain.c @@ -4,6 +4,7 @@ #define WIN32_LEAN_AND_MEAN #include +#include /* __argc, __wargv */ int WINAPI wWinMain( HINSTANCE hInstance, /* handle to current instance */ diff --git a/Programs/_freeze_module.c b/Programs/_freeze_module.c index dd90d92e512c0..316c70d2c7824 100644 --- a/Programs/_freeze_module.c +++ b/Programs/_freeze_module.c @@ -12,6 +12,7 @@ #include #include +#include // malloc() #include #include #ifndef MS_WINDOWS diff --git a/Programs/_testembed.c b/Programs/_testembed.c index fa418e276114a..b61fe3461bdc9 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -11,6 +11,7 @@ #include #include #include +#include // putenv() #include /********************************************************* diff --git a/Python/dtoa.c b/Python/dtoa.c index e629b296426f3..6c44f68965176 100644 --- a/Python/dtoa.c +++ b/Python/dtoa.c @@ -119,6 +119,7 @@ #include "Python.h" #include "pycore_dtoa.h" +#include // exit() /* if PY_NO_SHORT_FLOAT_REPR is defined, then don't even try to compile the following code */ diff --git a/Python/errors.c b/Python/errors.c index f072c21496592..519f2d459effd 100644 --- a/Python/errors.c +++ b/Python/errors.c @@ -15,12 +15,13 @@ extern char *strerror(int); #endif #endif +#include #ifdef MS_WINDOWS -#include -#include +# include +# include +# include // _sys_nerr #endif -#include #ifdef __cplusplus extern "C" { diff --git a/Python/fileutils.c b/Python/fileutils.c index a36415946e218..173d34dd23f18 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -3,6 +3,7 @@ #include "pycore_runtime.h" // _PyRuntime #include "osdefs.h" // SEP #include +#include // mbstowcs() #ifdef MS_WINDOWS # include diff --git a/Python/initconfig.c b/Python/initconfig.c index ba6d19db20048..b0d54b0472fb0 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -10,7 +10,9 @@ #include "pycore_pystate.h" // _PyThreadState_GET() #include "osdefs.h" // DELIM + #include // setlocale() +#include // getenv() #if defined(MS_WINDOWS) || defined(__CYGWIN__) # ifdef HAVE_IO_H # include diff --git a/Python/preconfig.c b/Python/preconfig.c index d59273159a671..afa16cccf32e9 100644 --- a/Python/preconfig.c +++ b/Python/preconfig.c @@ -4,7 +4,9 @@ #include "pycore_initconfig.h" // _PyArgv #include "pycore_pymem.h" // _PyMem_GetAllocatorName() #include "pycore_runtime.h" // _PyRuntime_Initialize() + #include // setlocale() +#include // getenv() /* Forward declarations */ diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index c67a9b7fbfa2d..c5a209abae61a 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -16,6 +16,7 @@ #include "pycore_traceback.h" // _Py_DumpTracebackThreads() #include // setlocale() +#include // getenv() #if defined(__APPLE__) #include diff --git a/Python/pystrhex.c b/Python/pystrhex.c index f7fe3b6eb85d0..880af44ea0e94 100644 --- a/Python/pystrhex.c +++ b/Python/pystrhex.c @@ -1,8 +1,8 @@ /* Format bytes as hexadecimal */ #include "Python.h" -#include "pycore_strhex.h" // _Py_strhex_with_sep() - +#include "pycore_strhex.h" // _Py_strhex_with_sep() +#include // abs() static PyObject *_Py_strhex_impl(const char* argbuf, const Py_ssize_t arglen, const PyObject* sep, int bytes_per_sep_group, From webhook-mailer at python.org Wed Oct 13 13:59:54 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 17:59:54 -0000 Subject: [Python-checkins] bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) (GH-28934) Message-ID: https://github.com/python/cpython/commit/f40b230df91f29fa4a84c1d127b4eab56a6eda27 commit: f40b230df91f29fa4a84c1d127b4eab56a6eda27 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T19:59:45+02:00 summary: bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) (GH-28934) At import time, the xmlrpc.client module uses different date formats to test strftime so it can format years with 4 digits consistently. Depending on the underlying C library and its strftime implementation some of these calls can result in ValueErrors, blocking the xmlrpc.client module from being imported. This commit changes the behavior of this bit of code to react to ValueError exceptions, treating the format that caused them as an non-viable option. (cherry picked from commit 1c831353816ff699b54e804047a7242a09e98f5b) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst M Lib/xmlrpc/client.py diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py index 9e7449c88dfc0..a614cef6ab2f1 100644 --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -264,16 +264,22 @@ def __repr__(self): # Issue #13305: different format codes across platforms _day0 = datetime(1, 1, 1) -if _day0.strftime('%Y') == '0001': # Mac OS X +def _try(fmt): + try: + return _day0.strftime(fmt) == '0001' + except ValueError: + return False +if _try('%Y'): # Mac OS X def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S") -elif _day0.strftime('%4Y') == '0001': # Linux +elif _try('%4Y'): # Linux def _iso8601_format(value): return value.strftime("%4Y%m%dT%H:%M:%S") else: def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S").zfill(17) del _day0 +del _try def _strftime(value): diff --git a/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst new file mode 100644 index 0000000000000..eec77ceccf933 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst @@ -0,0 +1,3 @@ +Make :mod:`xmlrpc.client` more robust to C runtimes where the underlying C +``strftime`` function results in a ``ValueError`` when testing for year +formatting options. From webhook-mailer at python.org Wed Oct 13 14:00:11 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 18:00:11 -0000 Subject: [Python-checkins] bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) (GH-28935) Message-ID: https://github.com/python/cpython/commit/9210eff61b75edabbe9263df0c4a303fc2149a22 commit: 9210eff61b75edabbe9263df0c4a303fc2149a22 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T20:00:05+02:00 summary: bpo-45386: Handle strftime's ValueError graciously in xmlrpc.client (GH-28765) (GH-28935) At import time, the xmlrpc.client module uses different date formats to test strftime so it can format years with 4 digits consistently. Depending on the underlying C library and its strftime implementation some of these calls can result in ValueErrors, blocking the xmlrpc.client module from being imported. This commit changes the behavior of this bit of code to react to ValueError exceptions, treating the format that caused them as an non-viable option. (cherry picked from commit 1c831353816ff699b54e804047a7242a09e98f5b) Co-authored-by: rtobar files: A Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst M Lib/xmlrpc/client.py diff --git a/Lib/xmlrpc/client.py b/Lib/xmlrpc/client.py index 9e7449c88dfc0..a614cef6ab2f1 100644 --- a/Lib/xmlrpc/client.py +++ b/Lib/xmlrpc/client.py @@ -264,16 +264,22 @@ def __repr__(self): # Issue #13305: different format codes across platforms _day0 = datetime(1, 1, 1) -if _day0.strftime('%Y') == '0001': # Mac OS X +def _try(fmt): + try: + return _day0.strftime(fmt) == '0001' + except ValueError: + return False +if _try('%Y'): # Mac OS X def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S") -elif _day0.strftime('%4Y') == '0001': # Linux +elif _try('%4Y'): # Linux def _iso8601_format(value): return value.strftime("%4Y%m%dT%H:%M:%S") else: def _iso8601_format(value): return value.strftime("%Y%m%dT%H:%M:%S").zfill(17) del _day0 +del _try def _strftime(value): diff --git a/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst new file mode 100644 index 0000000000000..eec77ceccf933 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-07-00-05-05.bpo-45386.q9ORpA.rst @@ -0,0 +1,3 @@ +Make :mod:`xmlrpc.client` more robust to C runtimes where the underlying C +``strftime`` function results in a ``ValueError`` when testing for year +formatting options. From webhook-mailer at python.org Wed Oct 13 16:36:21 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 13 Oct 2021 20:36:21 -0000 Subject: [Python-checkins] [3.9] bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) (GH-28932) Message-ID: https://github.com/python/cpython/commit/fb7203453855a43907048ff2b8f9b890305b3d15 commit: fb7203453855a43907048ff2b8f9b890305b3d15 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-13T22:36:05+02:00 summary: [3.9] bpo-24444: fix an error in argparse help when help for an option is blank (GH-28050) (GH-28932) (cherry picked from commit 6fafc25aea8689048314b5bf7a9bb986bb1ce238) Co-authored-by: andrei kulakov Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst M Lib/argparse.py M Lib/test/test_argparse.py diff --git a/Lib/argparse.py b/Lib/argparse.py index ecc92778c515b..cb5345d555afe 100644 --- a/Lib/argparse.py +++ b/Lib/argparse.py @@ -526,12 +526,13 @@ def _format_action(self, action): parts = [action_header] # if there was help for the action, add lines of help text - if action.help: + if action.help and action.help.strip(): help_text = self._expand_help(action) - help_lines = self._split_lines(help_text, help_width) - parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) - for line in help_lines[1:]: - parts.append('%*s%s\n' % (help_position, '', line)) + if help_text: + help_lines = self._split_lines(help_text, help_width) + parts.append('%*s%s\n' % (indent_first, '', help_lines[0])) + for line in help_lines[1:]: + parts.append('%*s%s\n' % (help_position, '', line)) # or add a newline if the description doesn't end with one elif not action_header.endswith('\n'): diff --git a/Lib/test/test_argparse.py b/Lib/test/test_argparse.py index 82c1f6c60f828..c3c3a75fafff7 100644 --- a/Lib/test/test_argparse.py +++ b/Lib/test/test_argparse.py @@ -2143,6 +2143,44 @@ def test_help_non_breaking_spaces(self): wrap\N{NO-BREAK SPACE}at non-breaking spaces ''')) + def test_help_blank(self): + # Issue 24444 + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', + help=' ') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] foo + + main description + + positional arguments: + foo \ + + + optional arguments: + -h, --help show this help message and exit + ''')) + + parser = ErrorRaisingArgumentParser( + prog='PROG', description='main description') + parser.add_argument( + 'foo', choices=[], + help='%(choices)s') + self.assertEqual(parser.format_help(), textwrap.dedent('''\ + usage: PROG [-h] {} + + main description + + positional arguments: + {} \ + + + optional arguments: + -h, --help show this help message and exit + ''')) + def test_help_alternate_prefix_chars(self): parser = self._get_parser(prefix_chars='+:/') self.assertEqual(parser.format_usage(), diff --git a/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst new file mode 100644 index 0000000000000..efcacb8f0eb59 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-30-00-19-23.bpo-24444.Ki4bgz.rst @@ -0,0 +1,2 @@ +Fixed an error raised in :mod:`argparse` help display when help for an +option is set to 1+ blank spaces or when *choices* arg is an empty container. From webhook-mailer at python.org Wed Oct 13 17:27:58 2021 From: webhook-mailer at python.org (vstinner) Date: Wed, 13 Oct 2021 21:27:58 -0000 Subject: [Python-checkins] bpo-45440: Require math.h isinf() to build (GH-28894) Message-ID: https://github.com/python/cpython/commit/194a9526d8ee6abbbe58ef48520ec87a7e83f327 commit: 194a9526d8ee6abbbe58ef48520ec87a7e83f327 branch: main author: Victor Stinner committer: vstinner date: 2021-10-13T23:27:50+02:00 summary: bpo-45440: Require math.h isinf() to build (GH-28894) Building Python now requires a C99 header file providing isinf(), isnan() and isfinite() functions. Remove the Py_FORCE_DOUBLE() macro. It was used by the Py_IS_INFINITY() macro. Changes: * Remove Py_IS_NAN(), Py_IS_INFINITY() and Py_IS_FINITE() in PC/pyconfig.h. * Remove the _Py_force_double() function. * configure no longer checks if math.h defines isinf(), isnan() and isfinite(). files: A Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst A Misc/NEWS.d/next/C API/2021-10-12-02-13-41.bpo-45440.Gf94rE.rst M Doc/whatsnew/3.11.rst M Include/internal/pycore_pymath.h M Include/pymath.h M PC/pyconfig.h M Python/pymath.c M configure M configure.ac M pyconfig.h.in diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d18711d0b8df2..48d454d9aac99 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -460,6 +460,10 @@ Build Changes * libpython is no longer linked against libcrypt. (Contributed by Mike Gilbert in :issue:`45433`.) +* Building Python now requires a C99 ```` header file providing + ``isinf()``, ``isnan()`` and ``isfinite()`` functions. + (Contributed by Victor Stinner in :issue:`45440`.) + C API Changes ============= @@ -605,3 +609,7 @@ Removed * Remove the ``pystrhex.h`` header file. It only contains private functions. C extensions should only include the main ```` header file. (Contributed by Victor Stinner in :issue:`45434`.) + +* Remove the ``Py_FORCE_DOUBLE()`` macro. It was used by the + ``Py_IS_INFINITY()`` macro. + (Contributed by Victor Stinner in :issue:`45440`.) diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h index 38f76d0461a89..32743fc83f5c0 100644 --- a/Include/internal/pycore_pymath.h +++ b/Include/internal/pycore_pymath.h @@ -142,6 +142,9 @@ extern void _Py_set_387controlword(unsigned short); // Get and set x87 control word for VisualStudio/x86. // x87 is not supported in 64-bit or ARM. #if defined(_MSC_VER) && !defined(_WIN64) && !defined(_M_ARM) + +#include // __control87_2() + #define _Py_SET_53BIT_PRECISION_HEADER \ unsigned int old_387controlword, new_387controlword, out_387controlword // We use the __control87_2 function to set only the x87 control word. diff --git a/Include/pymath.h b/Include/pymath.h index d688e5033e0cc..57310fc097e73 100644 --- a/Include/pymath.h +++ b/Include/pymath.h @@ -4,8 +4,6 @@ #ifndef Py_PYMATH_H #define Py_PYMATH_H -#include "pyconfig.h" // HAVE_DECL_ISNAN - /* High precision definition of pi and e (Euler) * The values are taken from libc6's math.h. */ @@ -29,77 +27,17 @@ #define Py_MATH_TAU 6.2831853071795864769252867665590057683943L #endif +// Py_IS_NAN(X) +// Return 1 if float or double arg is a NaN, else 0. +#define Py_IS_NAN(X) isnan(X) -/* On x86, Py_FORCE_DOUBLE forces a floating-point number out of an x87 FPU - register and into a 64-bit memory location, rounding from extended - precision to double precision in the process. On other platforms it does - nothing. */ - -/* we take double rounding as evidence of x87 usage */ -#ifndef Py_LIMITED_API -#ifndef Py_FORCE_DOUBLE -# ifdef X87_DOUBLE_ROUNDING -PyAPI_FUNC(double) _Py_force_double(double); -# define Py_FORCE_DOUBLE(X) (_Py_force_double(X)) -# else -# define Py_FORCE_DOUBLE(X) (X) -# endif -#endif -#endif - -/* Py_IS_NAN(X) - * Return 1 if float or double arg is a NaN, else 0. - * Caution: - * X is evaluated more than once. - * This may not work on all platforms. Each platform has *some* - * way to spell this, though -- override in pyconfig.h if you have - * a platform where it doesn't work. - * Note: PC/pyconfig.h defines Py_IS_NAN as _isnan - */ -#ifndef Py_IS_NAN -# if defined HAVE_DECL_ISNAN && HAVE_DECL_ISNAN == 1 -# define Py_IS_NAN(X) isnan(X) -# else -# define Py_IS_NAN(X) ((X) != (X)) -# endif -#endif - -/* Py_IS_INFINITY(X) - * Return 1 if float or double arg is an infinity, else 0. - * Caution: - * X is evaluated more than once. - * This implementation may set the underflow flag if |X| is very small; - * it really can't be implemented correctly (& easily) before C99. - * Override in pyconfig.h if you have a better spelling on your platform. - * Py_FORCE_DOUBLE is used to avoid getting false negatives from a - * non-infinite value v sitting in an 80-bit x87 register such that - * v becomes infinite when spilled from the register to 64-bit memory. - * Note: PC/pyconfig.h defines Py_IS_INFINITY as _isinf - */ -#ifndef Py_IS_INFINITY -# if defined HAVE_DECL_ISINF && HAVE_DECL_ISINF == 1 -# define Py_IS_INFINITY(X) isinf(X) -# else -# define Py_IS_INFINITY(X) ((X) && \ - (Py_FORCE_DOUBLE(X)*0.5 == Py_FORCE_DOUBLE(X))) -# endif -#endif +// Py_IS_INFINITY(X) +// Return 1 if float or double arg is an infinity, else 0. +#define Py_IS_INFINITY(X) isinf(X) -/* Py_IS_FINITE(X) - * Return 1 if float or double arg is neither infinite nor NAN, else 0. - * Some compilers (e.g. VisualStudio) have intrinsics for this, so a special - * macro for this particular test is useful - * Note: PC/pyconfig.h defines Py_IS_FINITE as _finite - */ -#ifndef Py_IS_FINITE -# if defined HAVE_DECL_ISFINITE && HAVE_DECL_ISFINITE == 1 -# define Py_IS_FINITE(X) isfinite(X) -# elif defined HAVE_FINITE -# define Py_IS_FINITE(X) finite(X) -# else -# define Py_IS_FINITE(X) (!Py_IS_INFINITY(X) && !Py_IS_NAN(X)) -# endif -#endif +// Py_IS_FINITE(X) +// Return 1 if float or double arg is neither infinite nor NAN, else 0. +#define Py_IS_FINITE(X) isfinite(X) /* HUGE_VAL is supposed to expand to a positive double infinity. Python * uses Py_HUGE_VAL instead because some platforms are broken in this diff --git a/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst b/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst new file mode 100644 index 0000000000000..1c7c413155f45 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst @@ -0,0 +1,2 @@ +Building Python now requires a C99 ```` header file providing +``isinf()``, ``isnan()`` and ``isfinite()`` functions. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-10-12-02-13-41.bpo-45440.Gf94rE.rst b/Misc/NEWS.d/next/C API/2021-10-12-02-13-41.bpo-45440.Gf94rE.rst new file mode 100644 index 0000000000000..d9d695fd2ace4 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-12-02-13-41.bpo-45440.Gf94rE.rst @@ -0,0 +1,2 @@ +Remove the ``Py_FORCE_DOUBLE()`` macro. It was used by the ``Py_IS_INFINITY()`` +macro. Patch by Victor Stinner. diff --git a/PC/pyconfig.h b/PC/pyconfig.h index 012a89aab2bc8..75397772280dc 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -189,11 +189,6 @@ typedef _W64 int Py_ssize_t; typedef int pid_t; -#include -#define Py_IS_NAN _isnan -#define Py_IS_INFINITY(X) (!_finite(X) && !_isnan(X)) -#define Py_IS_FINITE(X) _finite(X) - /* define some ANSI types that are not defined in earlier Win headers */ #if _MSC_VER >= 1200 /* This file only exists in VC 6.0 or higher */ @@ -358,15 +353,9 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */ /* Define to 1 if you have the `round' function. */ #if _MSC_VER >= 1800 -#define HAVE_ROUND 1 +# define HAVE_ROUND 1 #endif -/* Define to 1 if you have the `isinf' macro. */ -#define HAVE_DECL_ISINF 1 - -/* Define to 1 if you have the `isnan' function. */ -#define HAVE_DECL_ISNAN 1 - /* Define if on AIX 3. System headers sometimes define this. We just want to avoid a redefinition error message. */ diff --git a/Python/pymath.c b/Python/pymath.c index d3c52a09650c5..b2681f2acc1f0 100644 --- a/Python/pymath.c +++ b/Python/pymath.c @@ -1,18 +1,5 @@ #include "Python.h" -#ifdef X87_DOUBLE_ROUNDING -/* On x86 platforms using an x87 FPU, this function is called from the - Py_FORCE_DOUBLE macro (defined in pymath.h) to force a floating-point - number out of an 80-bit x87 FPU register and into a 64-bit memory location, - thus rounding from extended precision to double precision. */ -double _Py_force_double(double x) -{ - volatile double y; - y = x; - return y; -} -#endif - #ifdef HAVE_GCC_ASM_FOR_X87 // Inline assembly for getting and setting the 387 FPU control word on diff --git a/configure b/configure index b2d1a64206cd4..c56ff1dc5cd6a 100755 --- a/configure +++ b/configure @@ -15090,40 +15090,6 @@ _ACEOF fi done -ac_fn_c_check_decl "$LINENO" "isinf" "ac_cv_have_decl_isinf" "#include -" -if test "x$ac_cv_have_decl_isinf" = xyes; then : - ac_have_decl=1 -else - ac_have_decl=0 -fi - -cat >>confdefs.h <<_ACEOF -#define HAVE_DECL_ISINF $ac_have_decl -_ACEOF -ac_fn_c_check_decl "$LINENO" "isnan" "ac_cv_have_decl_isnan" "#include -" -if test "x$ac_cv_have_decl_isnan" = xyes; then : - ac_have_decl=1 -else - ac_have_decl=0 -fi - -cat >>confdefs.h <<_ACEOF -#define HAVE_DECL_ISNAN $ac_have_decl -_ACEOF -ac_fn_c_check_decl "$LINENO" "isfinite" "ac_cv_have_decl_isfinite" "#include -" -if test "x$ac_cv_have_decl_isfinite" = xyes; then : - ac_have_decl=1 -else - ac_have_decl=0 -fi - -cat >>confdefs.h <<_ACEOF -#define HAVE_DECL_ISFINITE $ac_have_decl -_ACEOF - # For multiprocessing module, check that sem_open # actually works. For FreeBSD versions <= 7.2, diff --git a/configure.ac b/configure.ac index a6c6d1c49596a..1d0c753efac1a 100644 --- a/configure.ac +++ b/configure.ac @@ -4677,7 +4677,6 @@ LIBS="$LIBS $LIBM" AC_CHECK_FUNCS([acosh asinh atanh copysign erf erfc expm1 finite gamma]) AC_CHECK_FUNCS([hypot lgamma log1p log2 round tgamma]) -AC_CHECK_DECLS([isinf, isnan, isfinite], [], [], [[#include ]]) # For multiprocessing module, check that sem_open # actually works. For FreeBSD versions <= 7.2, diff --git a/pyconfig.h.in b/pyconfig.h.in index 3231cd68e063f..43a04e371a594 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -214,18 +214,6 @@ /* Define if you have the 'wchgat' function. */ #undef HAVE_CURSES_WCHGAT -/* Define to 1 if you have the declaration of `isfinite', and to 0 if you - don't. */ -#undef HAVE_DECL_ISFINITE - -/* Define to 1 if you have the declaration of `isinf', and to 0 if you don't. - */ -#undef HAVE_DECL_ISINF - -/* Define to 1 if you have the declaration of `isnan', and to 0 if you don't. - */ -#undef HAVE_DECL_ISNAN - /* Define to 1 if you have the declaration of `RTLD_DEEPBIND', and to 0 if you don't. */ #undef HAVE_DECL_RTLD_DEEPBIND From webhook-mailer at python.org Wed Oct 13 18:34:20 2021 From: webhook-mailer at python.org (brandtbucher) Date: Wed, 13 Oct 2021 22:34:20 -0000 Subject: [Python-checkins] Ensure that instruction cases are self-contained (GH-28938) Message-ID: https://github.com/python/cpython/commit/e71662c1ae817e728233ce93882c5b20f4c31ebc commit: e71662c1ae817e728233ce93882c5b20f4c31ebc branch: main author: Brandt Bucher committer: brandtbucher date: 2021-10-13T15:34:11-07:00 summary: Ensure that instruction cases are self-contained (GH-28938) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 0af233c0ba485..aef83b9c355fc 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1710,8 +1710,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } - /* We keep LOAD_CLOSURE so that the bytecode stays more readable. */ TARGET(LOAD_CLOSURE) { + /* We keep LOAD_CLOSURE so that the bytecode stays more readable. */ PyObject *value = GETLOCAL(oparg); if (value == NULL) { goto unbound_local_error; @@ -3858,10 +3858,10 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } -#define CANNOT_CATCH_MSG "catching classes that do not inherit from "\ - "BaseException is not allowed" - TARGET(JUMP_IF_NOT_EXC_MATCH) { + const char *cannot_catch_msg = "catching classes that do not " + "inherit from BaseException is not " + "allowed"; PyObject *right = POP(); PyObject *left = POP(); if (PyTuple_Check(right)) { @@ -3871,7 +3871,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *exc = PyTuple_GET_ITEM(right, i); if (!PyExceptionClass_Check(exc)) { _PyErr_SetString(tstate, PyExc_TypeError, - CANNOT_CATCH_MSG); + cannot_catch_msg); Py_DECREF(left); Py_DECREF(right); goto error; @@ -3881,7 +3881,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr else { if (!PyExceptionClass_Check(right)) { _PyErr_SetString(tstate, PyExc_TypeError, - CANNOT_CATCH_MSG); + cannot_catch_msg); Py_DECREF(left); Py_DECREF(right); goto error; From webhook-mailer at python.org Thu Oct 14 06:17:34 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 10:17:34 -0000 Subject: [Python-checkins] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) Message-ID: https://github.com/python/cpython/commit/c96d1546b11b4c282a7e21737cb1f5d16349656d commit: c96d1546b11b4c282a7e21737cb1f5d16349656d branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-14T13:17:00+03:00 summary: bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.unicode_escape_decode(). It is True by default to match the former behavior. files: A Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst M Include/cpython/unicodeobject.h M Lib/encodings/unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c M Parser/string_parser.c diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 0cbdbdbbe0af2..bc5a3b4bd0b99 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -777,12 +777,20 @@ PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF16( /* --- Unicode-Escape Codecs ---------------------------------------------- */ +/* Variant of PyUnicode_DecodeUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); /* Helper for PyUnicode_DecodeUnicodeEscape that detects invalid escape chars. */ -PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscape( +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeInternal( const char *string, /* Unicode-Escape encoded string */ Py_ssize_t length, /* size of string */ const char *errors, /* error handling */ + Py_ssize_t *consumed, /* bytes consumed */ const char **first_invalid_escape /* on return, points to first invalid escaped char in string. */ diff --git a/Lib/encodings/unicode_escape.py b/Lib/encodings/unicode_escape.py index 817f93265a463..9b1ce99b339ae 100644 --- a/Lib/encodings/unicode_escape.py +++ b/Lib/encodings/unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index f1a149f19b7d2..288a3006cdeb7 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -114,7 +114,7 @@ def check_partial(self, input, partialresults): q = Queue(b"") r = codecs.getreader(self.encoding)(q) result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): q.write(bytes([c])) result += r.read() self.assertEqual(result, partialresult) @@ -125,7 +125,7 @@ def check_partial(self, input, partialresults): # do the check again, this time using an incremental decoder d = codecs.getincrementaldecoder(self.encoding)() result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): result += d.decode(bytes([c])) self.assertEqual(result, partialresult) # check that there's nothing left in the buffers @@ -135,7 +135,7 @@ def check_partial(self, input, partialresults): # Check whether the reset method works properly d.reset() result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): result += d.decode(bytes([c])) self.assertEqual(result, partialresult) # check that there's nothing left in the buffers @@ -2353,7 +2353,11 @@ def test_unicode_escape(self): (r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10)) -class UnicodeEscapeTest(unittest.TestCase): +class UnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.unicode_escape_decode(b""), ("", 0)) @@ -2440,6 +2444,44 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '', + '', + '', + '\x00', + '\x00', + '\x00\t', + '\x00\t', + '\x00\t\n', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) class RawUnicodeEscapeTest(unittest.TestCase): def test_empty(self): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst new file mode 100644 index 0000000000000..c1c4ed1ace248 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index 2e8cb97fe77c9..fc74127ce56c4 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -489,17 +489,20 @@ _codecs_utf_32_ex_decode_impl(PyObject *module, Py_buffer *data, _codecs.unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=3ca3c917176b82ab input=8328081a3a569bd6]*/ + const char *errors, int final) +/*[clinic end generated code: output=b284f97b12c635ee input=6154f039a9f7c639]*/ { - PyObject *decoded = PyUnicode_DecodeUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index 43378f94f9845..a7086dd6e18d7 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1063,7 +1063,7 @@ _codecs_utf_32_ex_decode(PyObject *module, PyObject *const *args, Py_ssize_t nar } PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, -"unicode_escape_decode($module, data, errors=None, /)\n" +"unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1072,7 +1072,7 @@ PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1080,8 +1080,9 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1122,8 +1123,15 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ _PyArg_BadArgument("unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2801,4 +2809,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=557c3b37e4c492ac input=a9049054013a1b77]*/ +/*[clinic end generated code: output=9e9fb1d5d81577e0 input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 741cf9dd13d9f..af3b3335d60ca 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6342,9 +6342,10 @@ PyUnicode_AsUTF16String(PyObject *unicode) static _PyUnicode_Name_CAPI *ucnhash_capi = NULL; PyObject * -_PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeInternal(const char *s, Py_ssize_t size, const char *errors, + Py_ssize_t *consumed, const char **first_invalid_escape) { const char *starts = s; @@ -6357,6 +6358,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, *first_invalid_escape = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } /* Escaped strings will always be longer than the resulting @@ -6407,7 +6411,7 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; - goto error; + goto incomplete; } c = (unsigned char) *s++; @@ -6461,7 +6465,10 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, count = 8; message = "truncated \\UXXXXXXXX escape"; hexescape: - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6474,12 +6481,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (count) { - goto error; - } /* when we get here, ch is a 32-bit unicode character */ if (ch > MAX_UNICODE) { @@ -6506,14 +6510,20 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } message = "malformed \\N character escape"; - if (s < end && *s == '{') { + if (s >= end) { + goto incomplete; + } + if (*s == '{') { const char *start = ++s; size_t namelen; /* look for the closing brace */ while (s < end && *s != '}') s++; + if (s >= end) { + goto incomplete; + } namelen = s - start; - if (namelen && s < end) { + if (namelen) { /* found a name. look it up in the unicode database */ s++; ch = 0xffffffff; /* in case 'getcode' messes up */ @@ -6539,6 +6549,11 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, continue; } + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } error: endinpos = s-starts; writer.min_length = end - s + writer.pos; @@ -6567,12 +6582,14 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } PyObject * -PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeStateful(const char *s, Py_ssize_t size, - const char *errors) + const char *errors, + Py_ssize_t *consumed) { const char *first_invalid_escape; - PyObject *result = _PyUnicode_DecodeUnicodeEscape(s, size, errors, + PyObject *result = _PyUnicode_DecodeUnicodeEscapeInternal(s, size, errors, + consumed, &first_invalid_escape); if (result == NULL) return NULL; @@ -6587,6 +6604,14 @@ PyUnicode_DecodeUnicodeEscape(const char *s, return result; } +PyObject * +PyUnicode_DecodeUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeUnicodeEscapeStateful(s, size, errors, NULL); +} + /* Return a Unicode-Escape string version of the Unicode object. */ PyObject * diff --git a/Parser/string_parser.c b/Parser/string_parser.c index cffe24e992e44..c6fe99c885d69 100644 --- a/Parser/string_parser.c +++ b/Parser/string_parser.c @@ -115,7 +115,7 @@ decode_unicode_with_escapes(Parser *parser, const char *s, size_t len, Token *t) s = buf; const char *first_invalid_escape; - v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape); + v = _PyUnicode_DecodeUnicodeEscapeInternal(s, len, NULL, NULL, &first_invalid_escape); if (v != NULL && first_invalid_escape != NULL) { if (warn_invalid_escape_sequence(parser, *first_invalid_escape, t) < 0) { From webhook-mailer at python.org Thu Oct 14 10:56:39 2021 From: webhook-mailer at python.org (markshannon) Date: Thu, 14 Oct 2021 14:56:39 -0000 Subject: [Python-checkins] bpo-45367: Specialize BINARY_MULTIPLY (GH-28727) Message-ID: https://github.com/python/cpython/commit/3b3d30e8f78271a488965c9cd11136e1aa890757 commit: 3b3d30e8f78271a488965c9cd11136e1aa890757 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: markshannon date: 2021-10-14T15:56:33+01:00 summary: bpo-45367: Specialize BINARY_MULTIPLY (GH-28727) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-05-03-49-07.bpo-45367._astoU.rst M Include/internal/pycore_code.h M Include/internal/pycore_long.h M Include/opcode.h M Lib/opcode.py M Objects/longobject.c M Python/ceval.c M Python/opcode_targets.h M Python/specialize.c diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 0b127ed28993b..a91209b39f654 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -307,7 +307,8 @@ int _Py_Specialize_StoreAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *nam int _Py_Specialize_LoadGlobal(PyObject *globals, PyObject *builtins, _Py_CODEUNIT *instr, PyObject *name, SpecializedCacheEntry *cache); int _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, SpecializedCacheEntry *cache); int _Py_Specialize_BinarySubscr(PyObject *sub, PyObject *container, _Py_CODEUNIT *instr); -int _Py_Specialize_BinaryAdd(PyObject *sub, PyObject *container, _Py_CODEUNIT *instr); +int _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr); +int _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *instr); #define PRINT_SPECIALIZATION_STATS 0 #define PRINT_SPECIALIZATION_STATS_DETAILED 0 diff --git a/Include/internal/pycore_long.h b/Include/internal/pycore_long.h index 7336c317c3f2c..8edc90218547f 100644 --- a/Include/internal/pycore_long.h +++ b/Include/internal/pycore_long.h @@ -35,6 +35,7 @@ static inline PyObject* _PyLong_GetOne(void) { return __PyLong_GetSmallInt_internal(1); } PyObject *_PyLong_Add(PyLongObject *left, PyLongObject *right); +PyObject *_PyLong_Multiply(PyLongObject *left, PyLongObject *right); #ifdef __cplusplus } diff --git a/Include/opcode.h b/Include/opcode.h index 15f722630dc5d..fabb8d123bb9c 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -141,33 +141,36 @@ extern "C" { #define BINARY_ADD_FLOAT 13 #define BINARY_ADD_UNICODE 14 #define BINARY_ADD_UNICODE_INPLACE_FAST 18 -#define BINARY_SUBSCR_ADAPTIVE 21 -#define BINARY_SUBSCR_LIST_INT 36 -#define BINARY_SUBSCR_TUPLE_INT 38 -#define BINARY_SUBSCR_DICT 39 -#define JUMP_ABSOLUTE_QUICK 40 -#define LOAD_ATTR_ADAPTIVE 41 -#define LOAD_ATTR_INSTANCE_VALUE 42 -#define LOAD_ATTR_WITH_HINT 43 -#define LOAD_ATTR_SLOT 44 -#define LOAD_ATTR_MODULE 45 -#define LOAD_GLOBAL_ADAPTIVE 46 -#define LOAD_GLOBAL_MODULE 47 -#define LOAD_GLOBAL_BUILTIN 48 -#define LOAD_METHOD_ADAPTIVE 58 -#define LOAD_METHOD_CACHED 80 -#define LOAD_METHOD_CLASS 81 -#define LOAD_METHOD_MODULE 87 -#define LOAD_METHOD_NO_DICT 88 -#define STORE_ATTR_ADAPTIVE 120 -#define STORE_ATTR_INSTANCE_VALUE 122 -#define STORE_ATTR_SLOT 123 -#define STORE_ATTR_WITH_HINT 127 -#define LOAD_FAST__LOAD_FAST 128 -#define STORE_FAST__LOAD_FAST 134 -#define LOAD_FAST__LOAD_CONST 140 -#define LOAD_CONST__LOAD_FAST 143 -#define STORE_FAST__STORE_FAST 149 +#define BINARY_MULTIPLY_ADAPTIVE 21 +#define BINARY_MULTIPLY_INT 36 +#define BINARY_MULTIPLY_FLOAT 38 +#define BINARY_SUBSCR_ADAPTIVE 39 +#define BINARY_SUBSCR_LIST_INT 40 +#define BINARY_SUBSCR_TUPLE_INT 41 +#define BINARY_SUBSCR_DICT 42 +#define JUMP_ABSOLUTE_QUICK 43 +#define LOAD_ATTR_ADAPTIVE 44 +#define LOAD_ATTR_INSTANCE_VALUE 45 +#define LOAD_ATTR_WITH_HINT 46 +#define LOAD_ATTR_SLOT 47 +#define LOAD_ATTR_MODULE 48 +#define LOAD_GLOBAL_ADAPTIVE 58 +#define LOAD_GLOBAL_MODULE 80 +#define LOAD_GLOBAL_BUILTIN 81 +#define LOAD_METHOD_ADAPTIVE 87 +#define LOAD_METHOD_CACHED 88 +#define LOAD_METHOD_CLASS 120 +#define LOAD_METHOD_MODULE 122 +#define LOAD_METHOD_NO_DICT 123 +#define STORE_ATTR_ADAPTIVE 127 +#define STORE_ATTR_INSTANCE_VALUE 128 +#define STORE_ATTR_SLOT 134 +#define STORE_ATTR_WITH_HINT 140 +#define LOAD_FAST__LOAD_FAST 143 +#define STORE_FAST__LOAD_FAST 149 +#define LOAD_FAST__LOAD_CONST 150 +#define LOAD_CONST__LOAD_FAST 151 +#define STORE_FAST__STORE_FAST 153 #define DO_TRACING 255 #ifdef NEED_OPCODE_JUMP_TABLES static uint32_t _PyOpcode_RelativeJump[8] = { diff --git a/Lib/opcode.py b/Lib/opcode.py index efd6aefccc571..9d9f35855e25b 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -225,6 +225,9 @@ def jabs_op(name, op): "BINARY_ADD_FLOAT", "BINARY_ADD_UNICODE", "BINARY_ADD_UNICODE_INPLACE_FAST", + "BINARY_MULTIPLY_ADAPTIVE", + "BINARY_MULTIPLY_INT", + "BINARY_MULTIPLY_FLOAT", "BINARY_SUBSCR_ADAPTIVE", "BINARY_SUBSCR_LIST_INT", "BINARY_SUBSCR_TUPLE_INT", diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-05-03-49-07.bpo-45367._astoU.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-05-03-49-07.bpo-45367._astoU.rst new file mode 100644 index 0000000000000..aae518da1e86f --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-05-03-49-07.bpo-45367._astoU.rst @@ -0,0 +1 @@ +Specialized the ``BINARY_MULTIPLY`` opcode to ``BINARY_MULTIPLY_INT`` and ``BINARY_MULTIPLY_FLOAT`` using the PEP 659 machinery. \ No newline at end of file diff --git a/Objects/longobject.c b/Objects/longobject.c index 111ef4566cda8..66e164974a92c 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -3594,13 +3594,11 @@ k_lopsided_mul(PyLongObject *a, PyLongObject *b) return NULL; } -static PyObject * -long_mul(PyLongObject *a, PyLongObject *b) +PyObject * +_PyLong_Multiply(PyLongObject *a, PyLongObject *b) { PyLongObject *z; - CHECK_BINOP(a, b); - /* fast path for single-digit multiplication */ if (IS_MEDIUM_VALUE(a) && IS_MEDIUM_VALUE(b)) { stwodigits v = medium_value(a) * medium_value(b); @@ -3617,6 +3615,13 @@ long_mul(PyLongObject *a, PyLongObject *b) return (PyObject *)z; } +static PyObject * +long_mul(PyLongObject *a, PyLongObject *b) +{ + CHECK_BINOP(a, b); + return _PyLong_Multiply(a, b); +} + /* Fast modulo division for single-digit longs. */ static PyObject * fast_mod(PyLongObject *a, PyLongObject *b) diff --git a/Python/ceval.c b/Python/ceval.c index aef83b9c355fc..de71ae5da0f82 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1933,14 +1933,73 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } TARGET(BINARY_MULTIPLY) { + PREDICTED(BINARY_MULTIPLY); + STAT_INC(BINARY_MULTIPLY, unquickened); PyObject *right = POP(); PyObject *left = TOP(); PyObject *res = PyNumber_Multiply(left, right); Py_DECREF(left); Py_DECREF(right); SET_TOP(res); - if (res == NULL) + if (res == NULL) { goto error; + } + DISPATCH(); + } + + TARGET(BINARY_MULTIPLY_ADAPTIVE) { + if (oparg == 0) { + PyObject *left = SECOND(); + PyObject *right = TOP(); + next_instr--; + if (_Py_Specialize_BinaryMultiply(left, right, next_instr) < 0) { + goto error; + } + DISPATCH(); + } + else { + STAT_INC(BINARY_MULTIPLY, deferred); + UPDATE_PREV_INSTR_OPARG(next_instr, oparg - 1); + STAT_DEC(BINARY_MULTIPLY, unquickened); + JUMP_TO_INSTRUCTION(BINARY_MULTIPLY); + } + } + + TARGET(BINARY_MULTIPLY_INT) { + PyObject *left = SECOND(); + PyObject *right = TOP(); + DEOPT_IF(!PyLong_CheckExact(left), BINARY_MULTIPLY); + DEOPT_IF(!PyLong_CheckExact(right), BINARY_MULTIPLY); + STAT_INC(BINARY_MULTIPLY, hit); + record_hit_inline(next_instr, oparg); + PyObject *prod = _PyLong_Multiply((PyLongObject *)left, (PyLongObject *)right); + SET_SECOND(prod); + Py_DECREF(right); + Py_DECREF(left); + STACK_SHRINK(1); + if (prod == NULL) { + goto error; + } + DISPATCH(); + } + + TARGET(BINARY_MULTIPLY_FLOAT) { + PyObject *left = SECOND(); + PyObject *right = TOP(); + DEOPT_IF(!PyFloat_CheckExact(left), BINARY_MULTIPLY); + DEOPT_IF(!PyFloat_CheckExact(right), BINARY_MULTIPLY); + STAT_INC(BINARY_MULTIPLY, hit); + record_hit_inline(next_instr, oparg); + double dprod = ((PyFloatObject *)left)->ob_fval * + ((PyFloatObject *)right)->ob_fval; + PyObject *prod = PyFloat_FromDouble(dprod); + SET_SECOND(prod); + Py_DECREF(right); + Py_DECREF(left); + STACK_SHRINK(1); + if (prod == NULL) { + goto error; + } DISPATCH(); } @@ -4954,6 +5013,7 @@ MISS_WITH_CACHE(LOAD_GLOBAL) MISS_WITH_CACHE(LOAD_METHOD) MISS_WITH_OPARG_COUNTER(BINARY_SUBSCR) MISS_WITH_OPARG_COUNTER(BINARY_ADD) +MISS_WITH_OPARG_COUNTER(BINARY_MULTIPLY) binary_subscr_dict_error: { diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 773f9254cf624..7d701e81a9a48 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -20,7 +20,7 @@ static void *opcode_targets[256] = { &&TARGET_BINARY_ADD_UNICODE_INPLACE_FAST, &&TARGET_BINARY_POWER, &&TARGET_BINARY_MULTIPLY, - &&TARGET_BINARY_SUBSCR_ADAPTIVE, + &&TARGET_BINARY_MULTIPLY_ADAPTIVE, &&TARGET_BINARY_MODULO, &&TARGET_BINARY_ADD, &&TARGET_BINARY_SUBTRACT, @@ -35,8 +35,11 @@ static void *opcode_targets[256] = { &&TARGET_MATCH_KEYS, &&TARGET_COPY_DICT_WITHOUT_KEYS, &&TARGET_PUSH_EXC_INFO, - &&TARGET_BINARY_SUBSCR_LIST_INT, + &&TARGET_BINARY_MULTIPLY_INT, &&TARGET_POP_EXCEPT_AND_RERAISE, + &&TARGET_BINARY_MULTIPLY_FLOAT, + &&TARGET_BINARY_SUBSCR_ADAPTIVE, + &&TARGET_BINARY_SUBSCR_LIST_INT, &&TARGET_BINARY_SUBSCR_TUPLE_INT, &&TARGET_BINARY_SUBSCR_DICT, &&TARGET_JUMP_ABSOLUTE_QUICK, @@ -45,9 +48,6 @@ static void *opcode_targets[256] = { &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LOAD_ATTR_SLOT, &&TARGET_LOAD_ATTR_MODULE, - &&TARGET_LOAD_GLOBAL_ADAPTIVE, - &&TARGET_LOAD_GLOBAL_MODULE, - &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, @@ -57,7 +57,7 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_ADD, &&TARGET_INPLACE_SUBTRACT, &&TARGET_INPLACE_MULTIPLY, - &&TARGET_LOAD_METHOD_ADAPTIVE, + &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_INPLACE_MODULO, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, @@ -79,15 +79,15 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_AND, &&TARGET_INPLACE_XOR, &&TARGET_INPLACE_OR, - &&TARGET_LOAD_METHOD_CACHED, - &&TARGET_LOAD_METHOD_CLASS, + &&TARGET_LOAD_GLOBAL_MODULE, + &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, &&TARGET_SETUP_ANNOTATIONS, &&TARGET_YIELD_VALUE, - &&TARGET_LOAD_METHOD_MODULE, - &&TARGET_LOAD_METHOD_NO_DICT, + &&TARGET_LOAD_METHOD_ADAPTIVE, + &&TARGET_LOAD_METHOD_CACHED, &&TARGET_POP_EXCEPT, &&TARGET_STORE_NAME, &&TARGET_DELETE_NAME, @@ -119,40 +119,40 @@ static void *opcode_targets[256] = { &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, - &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_LOAD_METHOD_CLASS, &&TARGET_JUMP_IF_NOT_EXC_MATCH, - &&TARGET_STORE_ATTR_INSTANCE_VALUE, - &&TARGET_STORE_ATTR_SLOT, + &&TARGET_LOAD_METHOD_MODULE, + &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, - &&TARGET_STORE_ATTR_WITH_HINT, - &&TARGET_LOAD_FAST__LOAD_FAST, + &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_STORE_ATTR_INSTANCE_VALUE, &&TARGET_GEN_START, &&TARGET_RAISE_VARARGS, &&TARGET_CALL_FUNCTION, &&TARGET_MAKE_FUNCTION, &&TARGET_BUILD_SLICE, - &&TARGET_STORE_FAST__LOAD_FAST, + &&TARGET_STORE_ATTR_SLOT, &&TARGET_MAKE_CELL, &&TARGET_LOAD_CLOSURE, &&TARGET_LOAD_DEREF, &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, - &&TARGET_LOAD_FAST__LOAD_CONST, + &&TARGET_STORE_ATTR_WITH_HINT, &&TARGET_CALL_FUNCTION_KW, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_LOAD_CONST__LOAD_FAST, + &&TARGET_LOAD_FAST__LOAD_FAST, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, - &&TARGET_STORE_FAST__STORE_FAST, - &&_unknown_opcode, - &&_unknown_opcode, + &&TARGET_STORE_FAST__LOAD_FAST, + &&TARGET_LOAD_FAST__LOAD_CONST, + &&TARGET_LOAD_CONST__LOAD_FAST, &&TARGET_MATCH_CLASS, - &&_unknown_opcode, + &&TARGET_STORE_FAST__STORE_FAST, &&_unknown_opcode, &&TARGET_FORMAT_VALUE, &&TARGET_BUILD_CONST_KEY_MAP, diff --git a/Python/specialize.c b/Python/specialize.c index 6efee7643a455..529eabf6bc3ab 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -124,6 +124,7 @@ _Py_GetSpecializationStats(void) { err += add_stat_dict(stats, LOAD_GLOBAL, "load_global"); err += add_stat_dict(stats, LOAD_METHOD, "load_method"); err += add_stat_dict(stats, BINARY_ADD, "binary_add"); + err += add_stat_dict(stats, BINARY_MULTIPLY, "binary_multiply"); err += add_stat_dict(stats, BINARY_SUBSCR, "binary_subscr"); err += add_stat_dict(stats, STORE_ATTR, "store_attr"); if (err < 0) { @@ -180,6 +181,7 @@ _Py_PrintSpecializationStats(void) print_stats(out, &_specialization_stats[LOAD_GLOBAL], "load_global"); print_stats(out, &_specialization_stats[LOAD_METHOD], "load_method"); print_stats(out, &_specialization_stats[BINARY_ADD], "binary_add"); + print_stats(out, &_specialization_stats[BINARY_MULTIPLY], "binary_multiply"); print_stats(out, &_specialization_stats[BINARY_SUBSCR], "binary_subscr"); print_stats(out, &_specialization_stats[STORE_ATTR], "store_attr"); if (out != stderr) { @@ -230,6 +232,7 @@ static uint8_t adaptive_opcodes[256] = { [LOAD_GLOBAL] = LOAD_GLOBAL_ADAPTIVE, [LOAD_METHOD] = LOAD_METHOD_ADAPTIVE, [BINARY_ADD] = BINARY_ADD_ADAPTIVE, + [BINARY_MULTIPLY] = BINARY_MULTIPLY_ADAPTIVE, [BINARY_SUBSCR] = BINARY_SUBSCR_ADAPTIVE, [STORE_ATTR] = STORE_ATTR_ADAPTIVE, }; @@ -240,6 +243,7 @@ static uint8_t cache_requirements[256] = { [LOAD_GLOBAL] = 2, /* _PyAdaptiveEntry and _PyLoadGlobalCache */ [LOAD_METHOD] = 3, /* _PyAdaptiveEntry, _PyAttrCache and _PyObjectCache */ [BINARY_ADD] = 0, + [BINARY_MULTIPLY] = 0, [BINARY_SUBSCR] = 0, [STORE_ATTR] = 2, /* _PyAdaptiveEntry and _PyAttrCache */ }; @@ -1188,3 +1192,32 @@ _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr) assert(!PyErr_Occurred()); return 0; } + +int +_Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *instr) +{ + if (!Py_IS_TYPE(left, Py_TYPE(right))) { + SPECIALIZATION_FAIL(BINARY_MULTIPLY, SPEC_FAIL_DIFFERENT_TYPES); + goto fail; + } + if (PyLong_CheckExact(left)) { + *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_INT, saturating_start()); + goto success; + } + else if (PyFloat_CheckExact(left)) { + *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_FLOAT, saturating_start()); + goto success; + } + else { + SPECIALIZATION_FAIL(BINARY_MULTIPLY, SPEC_FAIL_OTHER); + } +fail: + STAT_INC(BINARY_MULTIPLY, specialization_failure); + assert(!PyErr_Occurred()); + *instr = _Py_MAKECODEUNIT(_Py_OPCODE(*instr), ADAPTIVE_CACHE_BACKOFF); + return 0; +success: + STAT_INC(BINARY_MULTIPLY, specialization_success); + assert(!PyErr_Occurred()); + return 0; +} From webhook-mailer at python.org Thu Oct 14 12:00:06 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 14 Oct 2021 16:00:06 -0000 Subject: [Python-checkins] no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) Message-ID: https://github.com/python/cpython/commit/d413c503636cde2a6ab0ada25dccb0134633a8e6 commit: d413c503636cde2a6ab0ada25dccb0134633a8e6 branch: main author: Dong-hee Na committer: corona10 date: 2021-10-15T00:59:56+09:00 summary: no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) files: M Modules/expat/xmltok.c diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c index f2b6b406067ea..5b935718743ad 100644 --- a/Modules/expat/xmltok.c +++ b/Modules/expat/xmltok.c @@ -42,16 +42,16 @@ USE OR OTHER DEALINGS IN THE SOFTWARE. */ -#include -#include /* memcpy */ -#include - #ifdef _WIN32 # include "winconfig.h" #endif #include +#include +#include /* memcpy */ +#include + #include "expat_external.h" #include "internal.h" #include "xmltok.h" From webhook-mailer at python.org Thu Oct 14 12:36:03 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 14 Oct 2021 16:36:03 -0000 Subject: [Python-checkins] [3.10] no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) (GH-28951) Message-ID: https://github.com/python/cpython/commit/70b150a366e4d8e426f45d24a421fd70f833a8c5 commit: 70b150a366e4d8e426f45d24a421fd70f833a8c5 branch: 3.10 author: Dong-hee Na committer: corona10 date: 2021-10-15T01:35:53+09:00 summary: [3.10] no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) (GH-28951) files: M Modules/expat/xmltok.c diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c index f2b6b406067ea..5b935718743ad 100644 --- a/Modules/expat/xmltok.c +++ b/Modules/expat/xmltok.c @@ -42,16 +42,16 @@ USE OR OTHER DEALINGS IN THE SOFTWARE. */ -#include -#include /* memcpy */ -#include - #ifdef _WIN32 # include "winconfig.h" #endif #include +#include +#include /* memcpy */ +#include + #include "expat_external.h" #include "internal.h" #include "xmltok.h" From webhook-mailer at python.org Thu Oct 14 12:36:24 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 14 Oct 2021 16:36:24 -0000 Subject: [Python-checkins] no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) Message-ID: https://github.com/python/cpython/commit/38fadbc5b9fce409af4f40a093575535709a9bb0 commit: 38fadbc5b9fce409af4f40a093575535709a9bb0 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-14T09:36:19-07:00 summary: no-issue: Make silence about warning '_POSIX_C_SOURCE redefined' (GH-28948) (cherry picked from commit d413c503636cde2a6ab0ada25dccb0134633a8e6) Co-authored-by: Dong-hee Na files: M Modules/expat/xmltok.c diff --git a/Modules/expat/xmltok.c b/Modules/expat/xmltok.c index f2b6b406067ea..5b935718743ad 100644 --- a/Modules/expat/xmltok.c +++ b/Modules/expat/xmltok.c @@ -42,16 +42,16 @@ USE OR OTHER DEALINGS IN THE SOFTWARE. */ -#include -#include /* memcpy */ -#include - #ifdef _WIN32 # include "winconfig.h" #endif #include +#include +#include /* memcpy */ +#include + #include "expat_external.h" #include "internal.h" #include "xmltok.h" From webhook-mailer at python.org Thu Oct 14 13:02:34 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 17:02:34 -0000 Subject: [Python-checkins] [3.10] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28943) Message-ID: https://github.com/python/cpython/commit/0bff4ccbfd3297b0adf690655d3e9ddb0033bc69 commit: 0bff4ccbfd3297b0adf690655d3e9ddb0033bc69 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: serhiy-storchaka date: 2021-10-14T20:02:20+03:00 summary: [3.10] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28943) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.unicode_escape_decode(). It is True by default to match the former behavior. (cherry picked from commit c96d1546b11b4c282a7e21737cb1f5d16349656d) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst M Doc/data/python3.10.abi M Include/cpython/unicodeobject.h M Lib/encodings/unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c M Parser/string_parser.c diff --git a/Doc/data/python3.10.abi b/Doc/data/python3.10.abi index 1dca3e1c2e3de..0e678f3aedf9f 100644 --- a/Doc/data/python3.10.abi +++ b/Doc/data/python3.10.abi @@ -1046,6 +1046,7 @@ + @@ -1248,7 +1249,8 @@ - + + @@ -1647,333 +1649,1037 @@ + + + + - + - - + + - - - + + + - - - - + + + + - - - + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - - - - - + + + + + + + + - + - + - - - - + + + + - + - + - - - - - + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - - - + + + - + - + - + @@ -1998,12 +2704,12 @@ - + - - + + @@ -2021,7 +2727,7 @@ - + @@ -2033,7 +2739,7 @@ - + @@ -2057,7 +2763,7 @@ - + @@ -2075,7 +2781,7 @@ - + @@ -2117,12 +2823,12 @@ - + - - + + - + @@ -2137,13 +2843,13 @@ - + - + @@ -2164,7 +2870,7 @@ - + @@ -2179,12 +2885,12 @@ - - + + - - - + + + @@ -2195,22 +2901,22 @@ - + - + - + - + - + - + @@ -2225,10 +2931,10 @@ - + - + @@ -2236,16 +2942,16 @@ - - + + - + - - - + + + @@ -2254,16 +2960,16 @@ - + - + - + @@ -2278,61 +2984,61 @@ - + - - - - - - - + + + + + + + - + - - - - - + + + + + - + - + - + - + - + - + - - - - - + + + + + - + @@ -2374,10 +3080,10 @@ - + - + @@ -2407,13 +3113,13 @@ - + - + @@ -2422,24 +3128,24 @@ - + - + - + - + - + @@ -2452,59 +3158,107 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - + + - - - + + + - - - - + + + + - - - - + + + + - + - + - - - + + + - - - + + + - - - - + + + + + - - + @@ -2513,695 +3267,931 @@ - + - - + + - - - - - + + + + + - - - + + + - - - - + + + + - - - - + + + + - + - - + + - + - - + + - - - - + + + + - + - - + + - - - + + + - - + + - + - - + + - - + + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - - + + - - - - - - + + - - + + - - - - - - + + - - - + + + - - - + + + - - - + + + - - - + + + - - + + - - + + - - + + - - - + + + - + - + - + - - + + - + - + - - + + - - + + - - + + - - - + + + - - - - - - - - + + + - - - + + + - - - - + + + + - + - + - - + + - - + + - - - - + + + + - - - - - + + + + + - - - + + + - - - - + + + + - - - - + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - + + - - + + - - + + - - - + + + - - + + - - + + - - - + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - - - + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - - + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - + + - - - + + + - + - - - - - - + + + + + + - + - + - + - - - + + + - - + + - + - + - + - + - + - + - + - - - - - - - - + - + - - + + - + - - + + - + - - + + - + - + - - + + - + - - + + - - + + - + - + - - - + + + - - - - + + + + - - - - - - - - + + + - - + + - - + + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - - + - - + + - + + + + + + + + + + + + + + - + - + - + - - + + + + + + + + + + + + + + - + @@ -3226,42 +4216,237 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - + + + - - - + + + - + - - + + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -3269,22 +4454,22 @@ - + - + - + - + - + - + @@ -3292,253 +4477,298 @@ - + - - - - + + + + - - + + - - - - - - - - + + - - - + + + - + - + - - + + - - - - - - - - - - - + + - - - + + + - - - + + + - - - + + + - + - + - + - - + + - + - + - + - + - - + + - - - - + + + + - + - + - + - + - + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - + - + - - + + - + - + - - + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -3547,91 +4777,91 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -3639,7 +4869,7 @@ - + @@ -3651,13 +4881,13 @@ - + - + @@ -3666,73 +4896,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -3741,7 +4971,7 @@ - + @@ -3789,16 +5019,16 @@ - + - + - + - + @@ -3810,13 +5040,13 @@ - + - + @@ -3831,7 +5061,7 @@ - + @@ -3861,10 +5091,10 @@ - + - + @@ -3876,7 +5106,7 @@ - + @@ -3887,10 +5117,10 @@ - + - + @@ -3899,7 +5129,7 @@ - + @@ -3910,7 +5140,7 @@ - + @@ -3930,7 +5160,7 @@ - + @@ -3939,7 +5169,7 @@ - + @@ -3976,10 +5206,10 @@ - + - + @@ -3996,20 +5226,20 @@ - + - + - - + + - + @@ -4022,24 +5252,24 @@ - + - + - + - + - + - + @@ -4064,13 +5294,13 @@ - + - + @@ -4084,44 +5314,44 @@ - + - + - + - + - + - + - + - + - + - + - + - + @@ -4134,7 +5364,7 @@ - + @@ -4146,13 +5376,13 @@ - + - + @@ -4172,16 +5402,16 @@ - + - + - + @@ -4189,27 +5419,27 @@ - + - + - + - + - + - + @@ -4224,19 +5454,19 @@ - + - + - + - + - + @@ -4244,10 +5474,10 @@ - + - + @@ -4258,69 +5488,69 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -4332,7 +5562,7 @@ - + @@ -4347,43 +5577,43 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -4395,7 +5625,7 @@ - + @@ -4410,7 +5640,7 @@ - + @@ -4434,7 +5664,7 @@ - + @@ -4446,20 +5676,20 @@ - + - + - + - + - + @@ -4472,7 +5702,7 @@ - + @@ -4481,22 +5711,22 @@ - + - + - + - + - + @@ -4505,16 +5735,16 @@ - + - + - + - + @@ -4529,67 +5759,67 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -4598,7 +5828,7 @@ - + @@ -4614,24 +5844,24 @@ - + - + - + - + - - + + @@ -4639,22 +5869,22 @@ - - + + - + - + - + - + @@ -4662,21 +5892,21 @@ - + - + - + - + - + @@ -4686,12 +5916,12 @@ - + - + @@ -4699,7 +5929,7 @@ - + @@ -4712,12 +5942,12 @@ - + - + @@ -4726,15 +5956,15 @@ - + - - + + @@ -4742,7 +5972,7 @@ - + @@ -4766,7 +5996,7 @@ - + @@ -4774,7 +6004,7 @@ - + @@ -4782,7 +6012,7 @@ - + @@ -4793,16 +6023,16 @@ - + - + - + - + @@ -4817,7 +6047,7 @@ - + @@ -4827,12 +6057,12 @@ - + - - + + @@ -4840,25 +6070,25 @@ - + - + - + - + @@ -4866,24 +6096,24 @@ - + - + - + - + - + @@ -4899,12 +6129,12 @@ - + - + @@ -4912,7 +6142,7 @@ - + @@ -4920,27 +6150,27 @@ - + - + - + - + @@ -4948,7 +6178,7 @@ - + @@ -4956,33 +6186,33 @@ - + - + - + - + - + - + - + - + @@ -4992,709 +6222,709 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -5704,205 +6934,238 @@ - + - + - + - + - - - - - + + + + + - - - + + + - - - - + + + + - + - - + + - - - - - - - - - - + + + + - - - - + + + + - - - - + + + + - + - - - + + + - + - - + + - + - - + + - - + + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - - + + + - - + + - + - - + + - - - + + + - + - + - - - + + + - - - + + + - - + + - + - + - + - - - - - - + - + - + + + + + + - - - + + + - - + + - - + + + + + @@ -5911,13 +7174,13 @@ - + - + - + @@ -5925,246 +7188,303 @@ - - - - - - - - - - - + + - - + + - - + + - - - - - - - - + + + + - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - - + + - + - + - + - - - - - + + + + + - - - + + + - - + + - - + + - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -6179,25 +7499,25 @@ - + - + - + - + - + @@ -6232,6 +7552,48 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -6242,28 +7604,28 @@ - - - - + + + + - - - + + + - - - + + + - + - + @@ -6272,312 +7634,407 @@ - + - + - - - - - + + + + + - - - - + + + + - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - + + + - + - - - + + + - + - - - + + + - + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - + - + - - - - - - - - - - - - - - - - + - + - + - + - + - + - - - - - + - - - - - - - - - - - - - - + + - + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + @@ -6590,100 +8047,145 @@ - - - + + + - - - + + + - + - - - + + + - + + + + + - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + - - + + - - + + - + - + - - - + + + - - + + - - + + - - - - + + + + - - - + + + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + @@ -6692,55 +8194,51 @@ - + - + - + - - + + - - - - + + + + - - + + - - - - - + - - + + - + - + @@ -6748,6 +8246,82 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -6761,20 +8335,20 @@ - + - - + + - + - + @@ -6783,21 +8357,54 @@ - - - - + + + + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -6805,71 +8412,80 @@ - - + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - - + + + - - + + - - + + - - + + - - + + - - - + + + - - - - + + + + + + + + + + + + + @@ -6879,7 +8495,7 @@ - + @@ -6891,13 +8507,13 @@ - + - + @@ -6906,73 +8522,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -6981,7 +8597,7 @@ - + @@ -7030,23 +8646,50 @@ - + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -7054,13 +8697,25 @@ - - - + + + - - + + + + + + + + + + + + + + @@ -7068,104 +8723,115 @@ - - + + - - + + - - + + - - + + - - - - - + + + + + - - - - + + + + - - - + + + - - - - + + + + - - - - + + + + - - - + + + - - + + - - + + - + + + + + + + + + + + + + + + + - + - + - - - + + + - - - + + + - + - + - + - - - - - + @@ -7173,36 +8839,36 @@ - - - + + + - - - + + + - - + + - - - + + + - + - + @@ -7211,13 +8877,13 @@ - + - + - + @@ -7241,155 +8907,174 @@ - - - - + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - + - + - - - - - + - + - - - - - - - - - - - + + + - - + + - - - + + + - - - + + + - + - - + + - - + + - + - - + + - - + + - - - - - - + + - + - + - + - + - - - - - - - - - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -7403,79 +9088,74 @@ - - - + + + - - - - - - - - + + + - - - + + + - + - + - - - - + + + + - + - - + + - - - + + + - + - + - - + + - + - - - + + + - + - + - + - + @@ -7485,169 +9165,216 @@ - + - - - - + + + + - - + + - - + + - - + + - - + + - - + + - - - - + + + + - - - - + + + + - - - + + + - - - - - - - - - - + + + + - + - - - - - - - - - - - - - - + - - + + - + - - - - + + + + - - - + + + - - - + + + - - - - - - - + - - - + + + - + - - - - - - - - - - - - + - - - + + + - - + + - - + + - - + + - + - - - + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + @@ -7657,103 +9384,157 @@ - - - + + + - - - - + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - + + - - - - - + - + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - + - - - - + + + + - - - + + + - - + + + + + - - + + - - + + - - + + - - + + @@ -7766,7 +9547,7 @@ - + @@ -7775,181 +9556,195 @@ - + - + - + - + - + - + - + - + - + - + - - - - - + - - - - - - + + - - - + + + - + - + - + - + - - - + + + - - + + - - + + - + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + + + + - - + + - + - + - - + + - - + + - - + + - - - + + + - - - - - - + + - + - - + + - + - + - + - - - + + + - + @@ -7958,29 +9753,29 @@ - + - + - + - + - + - + @@ -7989,273 +9784,342 @@ - + - - + + - + - + - - + + - - + + - - + + - - + + - + - - + + - - + + - - + + - - - - - - + + - - - - + + + + - - - - + + + + - - - - - + + + + + - - - + + + - - - - - + + + + + - - + + - + - - - - - - + + - + - - - - + + + + - - - + + + - + - + - - + + - + - - - + + + - + - - + + - + - + - - + + - - - - + + + + - + - - - - - - + - + - + - - - - + + + + - - - - + + + + - - + + - - + + - - - - - - - - - - - - - - + + - - + + - + - + - - + + - - + + - - + + - + - - + + - - + + - - + + - - - + + + - - + + - - + + - - + + - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -8264,10 +10128,10 @@ - + - + @@ -8277,37 +10141,33 @@ - - - - - + - + - + - - + + - + - + - + - + @@ -8324,44 +10184,44 @@ - - + + - + - + - + - + - - + + - + - + - + - + - + @@ -8376,242 +10236,311 @@ - + - + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + - + - + - + - + - - - - - - - - - + + + + + + + + + - - - + + + - - + + + + - - + + + - - - - - - - - - - - - - - + + - - - - - + + - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - + - + - - + + - + - - - + + + - + - - + + - - + + - - + + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + - - + + - + - + - - - + + + - - - + + + - - - + + + - - + + - - + + - - + + - - + + + + + + + + + + + - + - + - + - - + + - + - + - - + + - + - + - - + + - + - - - + + + - - - - + + + + - + @@ -8623,7 +10552,7 @@ - + @@ -8639,38 +10568,44 @@ - + - + - + - + - + - + - - + + - - - + + + - - - - + + + + - - + + + + + + + + @@ -8678,97 +10613,89 @@ - - + + - - - - + + + + - + - + - - + + - - - - + + + + - - - + + + - - - - - - + + - + + + + + - - - + + + + - - - - - - - - - + + + + - - - - - - - - + + + + - - - + + + - - - - + + + + - + - + - + @@ -8776,10 +10703,10 @@ - + - + @@ -8788,895 +10715,1184 @@ - + - + - - + + - - + + - - - + + + - - + + - + - - - - - - + - - - - + + + + - - - + + + - + - - - - - + - - - - + + + - + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + + - - + + - + - - + + - + - - + + - + - + - + - + - - + + - + - - + + - + - - + + - + - + - + - + - - + + - - - - + + + + - - - + + + - + - + - - - - + + + + - - - + + + - + - - - + + + - + - + - + - - - - + + + + - - + + - + - - + + - - + + - - - - - - + + - + - + - + - + - - - + + + - - - + + + - - + + - - + + - - - + + + - + - + - + - - - - - - - + - - + + - + - + - - - - + + + + - + - + - - - - + + + + - + - - - - - - - - + - + - - - - - - - - - + - - - + + + - - - + + + - + - + - - + + - + - + - + - + - - + + - - + + - - - + + + - - - - - - - - - - - - + - - + + - - + + - - + + - - - - - - - + + - - + + - - - - - - + + + + + + - - - - - - + + + + + + - + - + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - - - + + + - - - + + + - - - + + + @@ -9688,47 +11904,47 @@ - + - + - + - + - + - + - + - + - + - + - + @@ -9737,51 +11953,59 @@ - + - - - - - + + + + + + + + + + + + + - - + + - - + + - - - + + + - - - + + + - + - + - + @@ -9797,160 +12021,376 @@ - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + - - + + - - - - - - - + + + + + + + - + - - + + - - + + - + - - + + - + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + - - - + + + - - - - + + + + - - - - + + + + - - + + - - + + - + - + - - - + + + - - + + - - + + - - - + + + - - + + - + - + - + - + - + - + - + @@ -9959,16 +12399,16 @@ - + - + - + - + @@ -9982,32 +12422,32 @@ - + - - + + - - - + + + - - + + - - - + + + - - + + - + - + @@ -10016,91 +12456,91 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -10108,7 +12548,7 @@ - + @@ -10120,13 +12560,13 @@ - + - + @@ -10135,73 +12575,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -10210,7 +12650,7 @@ - + @@ -10257,16 +12697,16 @@ - + - + - + - + @@ -10278,13 +12718,13 @@ - + - + @@ -10299,7 +12739,7 @@ - + @@ -10315,290 +12755,489 @@ - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + - - - - - - - - - + + + + - - + + - + - + - - + + - - + + - + - + - + - + - - + + - + - + - + - + - + - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + - - + + - - - - - - + + - - - - - - - + + - + - + - + - + - + - - - - + + + + - - - - + + + + - + - + - + - + - + - + - + - + - - - + + + - - - + + + - + - + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + - - - + + + - - - - + + + + @@ -10639,10 +13278,10 @@ - + - + @@ -10656,16 +13295,16 @@ - + - + - + - + @@ -10788,7 +13427,7 @@ - + @@ -10814,10 +13453,10 @@ - + - + @@ -10834,16 +13473,16 @@ - + - + - + - + @@ -10854,16 +13493,16 @@ - + - + - + - + @@ -10995,10 +13634,10 @@ - + - + @@ -11008,7 +13647,7 @@ - + @@ -11109,10 +13748,10 @@ - + - + @@ -11129,14 +13768,14 @@ - + - + @@ -11170,18 +13809,18 @@ - + - + - - + + @@ -11199,10 +13838,10 @@ - + - + @@ -11216,23 +13855,23 @@ - + - + - + - + - + @@ -11242,7 +13881,7 @@ - + @@ -11261,8 +13900,8 @@ - - + + @@ -11331,7 +13970,7 @@ - + @@ -11393,7 +14032,7 @@ - + @@ -11437,10 +14076,10 @@ - + - + @@ -11458,7 +14097,7 @@ - + @@ -11473,10 +14112,10 @@ - + - + @@ -11501,16 +14140,16 @@ - + - + - + - + @@ -11562,10 +14201,10 @@ - + - + @@ -11575,7 +14214,7 @@ - + @@ -11607,10 +14246,10 @@ - + - + @@ -11635,7 +14274,7 @@ - + @@ -11664,10 +14303,10 @@ - + - + @@ -11681,16 +14320,16 @@ - + - + - + - + @@ -11717,7 +14356,7 @@ - + @@ -11737,10 +14376,10 @@ - + - + @@ -11754,23 +14393,23 @@ - + - + - + - + - + @@ -11783,7 +14422,7 @@ - + @@ -11795,15 +14434,15 @@ - + - + - + @@ -11828,7 +14467,7 @@ - + @@ -11838,7 +14477,7 @@ - + @@ -11868,901 +14507,312 @@ - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - + + + - - + + - + - - + + - - + + - - + + - + - - + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + - - + + - - - + + + - - - - - - + + + + + + - - - - + + + + - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - + - + - + - + - + - + - + - + + + + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - - - + + - - - - + + - - - - - - - - - + + - - + + - - + + - - + + - - - - - - - - - - - - - - - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + @@ -12774,13 +14824,13 @@ - + - + @@ -12789,73 +14839,73 @@ - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + - + @@ -12864,7 +14914,7 @@ - + @@ -12909,2762 +14959,5581 @@ - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + - - + + + + + + - - - - - - - - - - - - - - - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + - - - - - - - + + + - - - - - - - + + - - - - - + + - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - + + - - - + + - - - + + - - - + + + + + + - - - - - + + - - + + - - + + - - - - + + - - - - - + + - - - - + + - - - - + + - - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - + + - - - + + - - + + - - - + + + - - - - + + + + + + - - - - + + + + - - - + + + + - - - + + + + + + + + - - - + + + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - - - + + + + + + - - - - - - - + + + + + + - - - - - + + + + + + + - - - - - + + + + + + + - - - - - + + + + + + - - - - - + + + + + + - - - - - - - + + + + + + + - - - - - - - + + + + + + + - - - - + + + + + + + - - - - + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + - + - - + + + + - + - - - - + + + + + - - - - - - - - + + + + + - - - - + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + - - - - - + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + - - + + - + - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - + - + + + + + + - + - - - - - - + - - - - - - - - - - - + - - + + - - - + + + + + + + + + + + + + - + - - - - - - - + - - - - - - - + + + + + - - - - + + + - - - + + + + - - - + + + + + + + + + + + + + + + + - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + - - - - + + + + - - - - + + + + + - - - + + + + + + - - - + + - - - + + - - - - - + + + + - - - - + + + + - - - + + + - - - - + + + - - - + + - - + + - - - + + - - + + - - + + - - + + - - + + - - - + + - - - + + - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - + + - - + + - - + + + + - - + + - - + + - - - - - - - - - + + - - + + - - - + + + + + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + + + - - - - + + + + + + + - - - + + + + + - - - + + + + - - - - + + + + + - - - - - + + + + + - - - + + + - - - - + + + - - - + + + - - - - - - - - - + + + - - + + + + + - - + + - - + + - - + + + + - - + + + + + - - + + + + - - - + + + + - - - + + + + + - - - + + + + - - + + + - - + + + - - - - + + + - - - + + + - - - + + + - - - + + - - + + + - - - + + - - - + + + + + - - - - - + + - - + + - - + + - - - - - - + + - - + + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - + + - - - + + - - - + + - - + + - - - + + - - - + + + + - - - + + + + - - - - - - + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - + + + - - - + + + + + - - - - - - - - - - - - - - - - - - - - - + + + + + - - - - + + - - + + - - - - + + - - + + - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - - - + + + + + + + - - - - - + + + + + - - - - + + + + + - - - - + + + + + - - - - + + + + + + + - - - + + + + + + + - - - + + + + - - - - - - - + + + + - - - - - - + + + + - - - - - + + + + - - - + + + + - - - + + + + + - - - - - + + - - - - - + + + + + + + - - - - + + + + + + + + + - - - - - - + + + + + + - - - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + - + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - + - - - - - - - - - + + + + - - - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + + - - - + + + - - + + + - - - + + + - - + + - - - + + - - - + + - - + + - - - + + - - - - + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - + + - - + + - - - + + - - + + - - - + + - - - - + + - - - + + - - - + + - - + + - - - + + - - - - + + - - - + + - - - + + - - + + - - - - + + - - - - + + - - - + + - - - - - - - - - - - - + + - - - + + - - + + - - + + - - + + - - + + - - - - + + - - + + - - - - - - - + + - - - - - - + + - - - - - + + - - - + + - - - - - - - - - - - + + - - - - - + + - - - - - - - - - + + - - - - - - - - - + + - - - - - + + - - - - + + - - - - - - - - - - - - - - - - + + - - - - - - + + - - - - + + - - - - + + - - - - - - - - - - + + - - - + + - - - - + + - - - + + - - - + + - - - - - - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - + + - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + - - - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - + + - - - - + + - - - - + + - - - - - - - - + + - - + + - - + + - - - + + - - - - - - - + + - - - - - - - - - - - - - - + + - - - - + + - - + + - - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - + + - - + + - - + + - - + + - - + + - - - - + + - - - + + - - - - + + - - - - - - + + - - - - - - + + - - - + + - - - + + - - + + - - + + - - + + - - + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + - + + + + + + + + + + + + + - + + + + + + + + + + - + + + + + + + + + + - + + + + + + + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - - + + - - - - - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -15672,65 +20541,161 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + - + - + - + - + - + - + - + + + + + + + + + + + + + + + + + + + - + + + + - + - + - + - + - + - + diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index 0b5067f5d3f4b..a00489534607d 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -834,12 +834,20 @@ PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF16( /* --- Unicode-Escape Codecs ---------------------------------------------- */ +/* Variant of PyUnicode_DecodeUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); /* Helper for PyUnicode_DecodeUnicodeEscape that detects invalid escape chars. */ -PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscape( +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeInternal( const char *string, /* Unicode-Escape encoded string */ Py_ssize_t length, /* size of string */ const char *errors, /* error handling */ + Py_ssize_t *consumed, /* bytes consumed */ const char **first_invalid_escape /* on return, points to first invalid escaped char in string. */ diff --git a/Lib/encodings/unicode_escape.py b/Lib/encodings/unicode_escape.py index 817f93265a463..9b1ce99b339ae 100644 --- a/Lib/encodings/unicode_escape.py +++ b/Lib/encodings/unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 328a47b2e3766..153912b241a09 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -114,7 +114,7 @@ def check_partial(self, input, partialresults): q = Queue(b"") r = codecs.getreader(self.encoding)(q) result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): q.write(bytes([c])) result += r.read() self.assertEqual(result, partialresult) @@ -125,7 +125,7 @@ def check_partial(self, input, partialresults): # do the check again, this time using an incremental decoder d = codecs.getincrementaldecoder(self.encoding)() result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): result += d.decode(bytes([c])) self.assertEqual(result, partialresult) # check that there's nothing left in the buffers @@ -135,7 +135,7 @@ def check_partial(self, input, partialresults): # Check whether the reset method works properly d.reset() result = "" - for (c, partialresult) in zip(input.encode(self.encoding), partialresults): + for (c, partialresult) in zip(input.encode(self.encoding), partialresults, strict=True): result += d.decode(bytes([c])) self.assertEqual(result, partialresult) # check that there's nothing left in the buffers @@ -2341,7 +2341,11 @@ def test_unicode_escape(self): (r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10)) -class UnicodeEscapeTest(unittest.TestCase): +class UnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.unicode_escape_decode(b""), ("", 0)) @@ -2428,6 +2432,44 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '', + '', + '', + '\x00', + '\x00', + '\x00\t', + '\x00\t', + '\x00\t\n', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) class RawUnicodeEscapeTest(unittest.TestCase): def test_empty(self): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst new file mode 100644 index 0000000000000..c1c4ed1ace248 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index 2e8cb97fe77c9..fc74127ce56c4 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -489,17 +489,20 @@ _codecs_utf_32_ex_decode_impl(PyObject *module, Py_buffer *data, _codecs.unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=3ca3c917176b82ab input=8328081a3a569bd6]*/ + const char *errors, int final) +/*[clinic end generated code: output=b284f97b12c635ee input=6154f039a9f7c639]*/ { - PyObject *decoded = PyUnicode_DecodeUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index 43378f94f9845..a7086dd6e18d7 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1063,7 +1063,7 @@ _codecs_utf_32_ex_decode(PyObject *module, PyObject *const *args, Py_ssize_t nar } PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, -"unicode_escape_decode($module, data, errors=None, /)\n" +"unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1072,7 +1072,7 @@ PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1080,8 +1080,9 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1122,8 +1123,15 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ _PyArg_BadArgument("unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2801,4 +2809,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=557c3b37e4c492ac input=a9049054013a1b77]*/ +/*[clinic end generated code: output=9e9fb1d5d81577e0 input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 11a3a6d0a4dce..e660834b4788f 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6408,9 +6408,10 @@ PyUnicode_AsUTF16String(PyObject *unicode) static _PyUnicode_Name_CAPI *ucnhash_capi = NULL; PyObject * -_PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeInternal(const char *s, Py_ssize_t size, const char *errors, + Py_ssize_t *consumed, const char **first_invalid_escape) { const char *starts = s; @@ -6423,6 +6424,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, *first_invalid_escape = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } /* Escaped strings will always be longer than the resulting @@ -6473,7 +6477,7 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; - goto error; + goto incomplete; } c = (unsigned char) *s++; @@ -6527,7 +6531,10 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, count = 8; message = "truncated \\UXXXXXXXX escape"; hexescape: - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6540,12 +6547,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (count) { - goto error; - } /* when we get here, ch is a 32-bit unicode character */ if (ch > MAX_UNICODE) { @@ -6572,14 +6576,20 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } message = "malformed \\N character escape"; - if (s < end && *s == '{') { + if (s >= end) { + goto incomplete; + } + if (*s == '{') { const char *start = ++s; size_t namelen; /* look for the closing brace */ while (s < end && *s != '}') s++; + if (s >= end) { + goto incomplete; + } namelen = s - start; - if (namelen && s < end) { + if (namelen) { /* found a name. look it up in the unicode database */ s++; ch = 0xffffffff; /* in case 'getcode' messes up */ @@ -6605,6 +6615,11 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, continue; } + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } error: endinpos = s-starts; writer.min_length = end - s + writer.pos; @@ -6633,12 +6648,14 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } PyObject * -PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeStateful(const char *s, Py_ssize_t size, - const char *errors) + const char *errors, + Py_ssize_t *consumed) { const char *first_invalid_escape; - PyObject *result = _PyUnicode_DecodeUnicodeEscape(s, size, errors, + PyObject *result = _PyUnicode_DecodeUnicodeEscapeInternal(s, size, errors, + consumed, &first_invalid_escape); if (result == NULL) return NULL; @@ -6653,6 +6670,14 @@ PyUnicode_DecodeUnicodeEscape(const char *s, return result; } +PyObject * +PyUnicode_DecodeUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeUnicodeEscapeStateful(s, size, errors, NULL); +} + /* Return a Unicode-Escape string version of the Unicode object. */ PyObject * diff --git a/Parser/string_parser.c b/Parser/string_parser.c index fb37d37553a55..dcd298cb358ee 100644 --- a/Parser/string_parser.c +++ b/Parser/string_parser.c @@ -115,7 +115,7 @@ decode_unicode_with_escapes(Parser *parser, const char *s, size_t len, Token *t) s = buf; const char *first_invalid_escape; - v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape); + v = _PyUnicode_DecodeUnicodeEscapeInternal(s, len, NULL, NULL, &first_invalid_escape); if (v != NULL && first_invalid_escape != NULL) { if (warn_invalid_escape_sequence(parser, *first_invalid_escape, t) < 0) { From webhook-mailer at python.org Thu Oct 14 13:03:39 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 17:03:39 -0000 Subject: [Python-checkins] [3.9] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28945) Message-ID: https://github.com/python/cpython/commit/7c722e32bf582108680f49983cf01eaed710ddb9 commit: 7c722e32bf582108680f49983cf01eaed710ddb9 branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-14T20:03:29+03:00 summary: [3.9] bpo-45461: Fix IncrementalDecoder and StreamReader in the "unicode-escape" codec (GH-28939) (GH-28945) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.unicode_escape_decode(). It is True by default to match the former behavior. (cherry picked from commit c96d1546b11b4c282a7e21737cb1f5d16349656d) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst M Doc/data/python3.9.abi M Include/cpython/unicodeobject.h M Lib/encodings/unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c M Parser/pegen/parse_string.c M Python/ast.c diff --git a/Doc/data/python3.9.abi b/Doc/data/python3.9.abi index 228fefcd87005..3b202fa39b047 100644 --- a/Doc/data/python3.9.abi +++ b/Doc/data/python3.9.abi @@ -1290,7 +1290,8 @@ - + + @@ -1691,9 +1692,40 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - @@ -1741,7 +1773,7 @@ - + @@ -1758,6 +1790,35 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -1784,2684 +1845,6297 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + - + + + + - + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - + - + + + + + + + + - + - - - - - - - - - - + + + + + + + + + + + + + - + - + - + - + - + - - - - - - - - - - - - - + - + - + - + - + - - - - + - - - - + - + + + + + + + + + + + + - + - + - + - + + + + + + + + + + + + + + - + - - - - + - - + + - - + + - - + + - - + + + + - - + + - - + + - - + + + + - - + + - - + + - - + + - - + + - - + + - - + + + + - - + + - - + + + + + - - + + - - + + + + + + + + + + + + + + + + - - + + - - + + - - + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + + + - - + + - - + + + + + + + + + + + + - - + + - - + + - - + + - - + + + + - - + + - - + + + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - + + + - + + + + - + - - - - - - - - - - + + + + + + - + + + + - + + + + - + - - - - - - + - + - + + + + - + - - + + - - + + - - + + + + - - + + - - + + + + + + + + + + - - + + - - + + - - + + - - + + - + - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - + - + - + - + - + - - - - - - - - - - - - - - - - - - - + - - - - - - - - - + - - - - + - - - - - - - - - - - - - + - - + + - - + + + + - - + + - - - - - - - - - - - + + + + + + + + + - + - + - + - - - - - - - - - - + + + + + + - + + + + + + - + + + + + + + - + + + + + + + + + + + + + + - + + + + - + + + + - + + + + - + - + - + - + + + + - + + + + - + - + - + - - - - - - - - - - - - - - - - - - - - - + + - - + + - - + + - - + + - - - - - - - - + + - - + + - - + + - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + - - + + - - - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + - - - - + + - - + + - - - - + + - - + + - - + + - - - - + + - - - - + + - - + + - - + + - - - - + + - - + + - - - - - - - - - - - + + - - + + - - + + - - - - + + - - + + - - + + - - - - + + - + + + - + - + - - + + + + + + - + - - + + - - + + - - - - - - - - - - - + + - - + + - - + + - - - - + + - - + + - - - - + + - - + + - - - - + + - - + + - - - - + + - - + + - - + + - - - - + + - - + + + + + - - - + + - - - - + - + + + + - + + + + - + + + + - + - + - + - + - + - + - + - + - - - - - - - + + + + - + - - - - - - - + - + - + - + - + - + - - + + - - + + + + + + + + + + + - - + + - - + + - - - - - - + - + + + + + + - - - - - - - + + + + - - + + - - - + - + + + + + + + + + + + + + + + + + + + + - + - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - - - + + + + - - - - - - - + + + + + - - - - - - - + + + + + + - - + + + + - - - - - - + + + + + + - - - - + + + + + - - - - - + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + + - - - + - - - - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - - + + + - - - - - - + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + + + + + + + - + - + - + - - + + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - + - - - - + - + + + + + + - + - + - - - - - - - - - - - - - - - - - - + - - - - - - - - - - + + - + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - - - - + - + - + - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + - - - - - - - - - - - + + + + + + - - - - - - - - - + + + + + + - - - - - - - - + + + + + + - - - - - - - - + + + + - - - - - - - - + + - - - - - - - + + - - - - - - + + - - - - - - + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - - + + + + + + + + + + + + - - - - + + - - - - + + - - - - + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + - - + + + + + + + + + - - + + + + + + + - - - - + + + + + + + + + + + + - - - - - + + + - - - + + + - - - - + + + - - - - - + + + + - - - - + + + + - - - + + - - - - + + + - - - - + + + + - - - + + - - - + + + + - - - + + + - - + + + - - - - - - - + + - - + + - - - - - - - + + - - + + + - - - + + + + - - + + - - - + + + + - - - + + - - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + - - - - + + - - - - + + - - - + + + + + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - + + + + + + + + + + + + + - - - + + + + - - - + + - - - + + + + - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - + + - - - - - - + - - - - - - - - - - - - - - - + + - - - - - - + - - - - + - - - - + + - - - - - - + + - - - - - + + - - - - - - + + - - - - - + + + + + + - - - - - + + + - - - + + + - - - + + + + + + - - - + + - - - + + + - - - - + + + + - - - + + + - - - - - - - - - - - - - - - - - - - + + + + + + + + + - - - - - - - - - - - - - - - - - - + + - - - + + - - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + - - - + + + + + - - - - + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - + - - - - + + - - - - - + + - - - - - + + + - - + - - + - - - - - + + - - - + - - - + - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + - - - + + + + + + - - - - - - - + + + + + + + + + + + + + + + + - - - - - - + + - - - - - - + + - - - - + + - - - - - - - - - - - - - - - - - - - + + - - - + + - - - - + + - - - - + + + + + + - - - + + - - - - - - - - - - - - - - - - - + + + + - - - - + + - - - - - - + + + + - - - - - - + + - - - - - - + + + + - - - - - + + - - - - - + + + + + + + - - - - - + + - - - - + + - - - - + + - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + @@ -4469,7 +8143,7 @@ - + @@ -4583,8928 +8257,9732 @@ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - + - + - + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + + + + - + - - - - - - - + - - + + - - - - + + - - + + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - + - + - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - - + - - - - + - - - - + - - - - + - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - + - - - - + + - - + + - - - - - - - - - - + + - - + + - - + + - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - + - + - + - - - - + - - - - + + - - + + - - - - - - - - - + + + - + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - - - + + - - - - - - - + - + - - - - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + - - - - - - - - - + + - + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + - - + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + - - + + + + - - + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + - - - - - - - - - - - - - - - - - - - - - - - - - - + - - + + - + - - - - - - + - + - + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + + + + + + + + + + - - - - + + + + - - - - + + - - - - - + + - - - - + + + - - - - - - + + + - - - - - + + + + - - - - - - + + + + - - + + + + + + + + - - - - - + + + - - - - + + + + + + + + + - - - + + + + + + - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - + + - - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - - - + + - - - - - - - + + - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - + + - - - + + - - - - + + - - - + + - - - + + - - - - - - - - + + - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - - + + - - - + + - - - - - + + - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - + + - - - + + - - - + + - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - - + + - - - + + - - - - + + - - - - + + - - - - - - - - - - - - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - + + - - - - - - + + - - - - - - + + - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + - - - - - - + + - - - - - - - - - - + + - - - - - - - - + + - - - - - - - - + + - - - + + + + - - - + + - - - + + - - - + + + + - - - + + + + - - - + + + + + + + + + - - - + + + + + + + + + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + + + + + + - - - + + + - - + + + + + + + + + + + - - + + + - - + + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + - - - + + + + + + - - - + - - - - - - - - + + - - - + + - - - - - - - + + - - - - - - - - - - - - - - + + - - - - - + + - - - + + - - - - - + + - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + - - - - + + - - - + + - - - - + + - - - + + - - - - - - + + - - - + + - - - - + + - - - - - + + - - - - + + - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - + + + + - - - - - + + + + + + + + - - - - - + + + + + + - - - - + + - - + - - - - - - - - - - - - - - + + - - + + + + + + + + + + + + + + + + + - - + + - - - - - + + + - - - + + - - + + + + + + - - - + - - + - - - + + - - - - + + - - - - - + - - - - - - + + - - - + + - - - - - - - - + + - - - + + - - - - + + - - - + + - - - - + + - - - + + - - - - + + - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + - - - + + + + - - - + + + + - - - - - - - + + + + + + + + + + + + + - - + - - - - - - - + + - - - + + - - - + + - - - - - - - - - - - - - - - - + - - - - - + + - - - - + - - - + + - - - - + - - - - + + + + + + + + + + + + + + - - - - + + + + - - - + + + + + - - + + + - - - + + + + + + + + + + + + + - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + - - - + + + - - - + + - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + - - - - + + - - - - + + - - - - - - + + - - - - - + + - - - - - + + - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - - - - - + + - - - - - - + + - - - + + - - - + + + + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + - - - + + + + + - - - - + + + + + - - + + + - - + + + + - - + + + + - - + + + + - - + + + + - - + + + - - - - - - + + + - - + + - - - - - - - - - - - - - - - + + - - - + + + - - - + + + - - - + + - - - - + + - - - - + + + + + + - - - - + + + - - - + + - - - - + + - - - - - + + - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - - - + + - - - - - + + - - - + + + + + + + + + + + + + + - - - - + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + - - - - + + + + + + + + - - - - + + + + - - - - + + + + - - - - - + + - - - - + + - - - - - - - - + - - - + + - - - - + - - - + - - + + + - - + - - - - - - + + - - - - + + - - + - - - - - - - - - - - - - - - - - - + + - - - - - + + - - + + - - - - - - - - - - + + - - - - - + + - - - + + - - - + + - - - - - + + - - - - - - - - - + + + + - - - - - - - - - + + + - - - - - + + + + + - - - - - - - + + + + + - - + + + - - + + - - - + + + + - + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + - + + + + - + - + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + - - - - + + - - + + - - + + - - + + - - - - - - + + - - + + - - - - - - - - - - - - - - - - - - - - - + + + + + + + - - + + + + - - - - + + - - - - + + + + - - - - + + + + + - - - - + + + + + - - - + + - - - + + + - - + + + + - - + + + - - + - + + + + + + + - - - - - - + + + + + + - - - - - - - - - - - - - - - + + + - - + + + + - - - - - - - + + + + - - + + + + - - - - + + + + + + - + + - + + + + - - + + + + + - - + + + + - - + + + + + - - + + - - + + + + + + + + + + + - - + + + + - - - + + + - - - + + + + - - - + + + - - - + + + + - - - - - + + + - - - - - + + + + + - - - - - + + + + + + + + + + - - - + + + + + + + + - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + - - - + + - - - + + + + + + + + + + + + + + + + - - - + + + + - - - - - + + - - - + + + + + - - - - + + + - - - - + + + - - - - - - - - - + + + + + + + - - - + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - - - + + + + + - - - + + + + + - - - - - - - - - - - - + + + + + + + - - - - + + + + + + - - - - - + + + + + + - - - + + + + + + - - - + + + + + + + - - - + + + + + + + - - - + + + + + + + - - - + + + + + + + - - + + + + + - - + + + + - - - - - - - + + + + - - + + + + + - - - - - - - - - - - - - - - - - - - + + + + + + - - - - + + + + + - - - + + + + + - - - + + + + + - - - + + + + - - - - - - - - - - - - - - - - - - - - - - + + + + + - - - - - + - - + + - - + - - - - + + - - - - + + - - - + + + + - - - + + + + - - - + + + + - - - - - + + + + - - + - - - - - - - - - - - - - - - - + - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + - + - + - + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - + - + + + + + + + + + + + + + + + + - + - + - + - - + + + + + + + + + + + + + + + + + + + - - - - - - - - + + + + - - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + - - - - - - - - - - - - - + + + - - - - - + + + - - - - + + - + + - - + - + + - - - - - - + - + - - - - - + + + + + + - - - - + + + + + + + + + + + + + + - - - + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - + + + - - + + - - - - - + + - - - + + - - - - + + + - - - - + + + + + + - - - + + + + + + - - - + + + + + - - + + + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + - - - - - - - + + + + + - - - - - - + + - - - - - + + + + + + - - - - - - + + + - - - + + - - - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - + - - - - + + - - - - + + - - - - + + + + + + + + + + + + + + + + + + + - - - + + - - - - - - - - + + - - - - - + + - - - - + + - - + - - - - - + + - - - - + + - - - + + - - - + + - - + + + + - - - - - - - + + + - - - - + + - - - - + + - - - + + - - - + + - - - - + + + + + + + + + + + + + + + + + + + + - + - + - - - - + - + - - - - - - - - - + - - - - - - - + + + + + + + + + + + - - - - - + + + + + + + - - - - + + + + + + + - - - - + + + + + - - - + + + + + - - - - + + + - - - - + + + - - - - - + + + - - - - + + + - - - + + + + + - - + - - + + - - - - + + + + - - - + + + + - - - + + + + - - - - - - - - - - - - - + + + + - - - + + + + + - - - - + + + + - - - - + + + - - - - + + - - - + + - - - - + + + - - - - - + + + - - - - + + - - - + + + - - - + + + + + + + + - - + - - + - - - - + + - - - - - - + + - - + - - - + + - - - - - + + - - - - - + + - - - - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + - - + - - - - - + + - - - - + + - - - - + + - - - - - + + - - - - - - + + - - - - - + + - - - - - + + - - - + + + + + + + + - - - + + - - - + - - - + - - - + + + - - - - + + + + - - - - + + + + - - - - + + - - - - + + + - - - - + + + - - - + + - - - - + + + + + - - - - + + + + + - - - - - - + + - - - - - + - - - - - + + - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - + + - - - - - - - - - - - - + + - - - - - - + + - - - - + + - - - + + + + + + + - - - - - + + + + + - - - - - - + + + + + - - - - + + + + + - - - - - - + + + + + + + - - - - - + + + + + + + - - + + + - - - - - + + + - - + + + - - - + + + - - - - - + + + + - - - - - + + - - - + + + + + + + + - - - - + + + + + + - - - - - + + - - - - - + + + + + + + - - - - + + + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + - - - - + + - - - + + - - - - - + + - - - - - - + + - - - + + + - - - - - - + + + - - - - - + + + - - - - - - - + + + - - - - - - + + - - - + + - - - - - - + + - - - - - + + - - - - - - - + + - - - - - - + + - - - + + - - - - - + + - - - - + + - - - - - - - - - - - - - - - + + - - - - - - - - + + - - - - - + + + + + + + - - - - - + + + + + + + + + - - - - - - - + + + + + + + + + + - - - - - - + + + + - - - - - - + + + + - - - - - + + + - - + + + - - - - - + + + + + + + + + + + + + + + + + + + + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - + + - - - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + - - - - + + + - - - + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + - - - - - + + + - - - - - + + + - - - + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + - - - - - - + + + + - - - - - + + - - - - - + + + - - - - - - + + + - - - - - + + + + - - - + + + - - - + + + + - - - - + + + - - - - - + + + - - - - + + - - - - + + + - - - + + - - - - - - + + - - - + + + - - - - - + + - - - - - + + + + + - - - - + + + + + - - - + + + + + - - - + + + - - - - + + + - - - - + + + - - - - + + + - - - + - - + - - - - - - - + + - - - - - - + - - - - + + - - - - + + + + - - + + + + + + - - - + + + - - - - + - - - + + - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - + + - - - - + + - - - + + - - - + + - - - - - - - + - - - + + - - - - + + - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - + - - - + + - - - - + + - - - - - - - - - + + - - - - - - - - + + + + + + + + - - - - - - - - + + - - - - - + + - - - - - - + + - - - - - - + + - - - - + + - - - - - + + - - - - - - - - + + - - - - - - + + - - - + + - - - - - - - - - - - - - - - - - + + - - - - - - - - - + - - - + + - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - + + - - + + - - + + - - - + + - - + + - - + + - - + + - - - + + - - + + - - - + + - - + + - - - + - - - + - - - - - + + - - - + - - - - - + + - - - - + + - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - + + - - - - - + + - - - - + + - - - + + - - - - - + + - - - - + + - - + - - + + - - + + - - - - + + - - - - - + + - - + - - + - - + + - - + - - + - - + - + - + - + - - + + - - + - - + + - - + + + + + + + + - - - - - + + + + + + + + + + + + + + + + - - - + + + - - - + + + + - - - + + + - - - + + + - - - + + + + - - + + + + - - - + + + + + + + - - - - - + + + - - - - - + + + + + + + + + - - - - + + - - - - - + + - - - - - + + - - - - - + + + + + - - - - - + + - - - + - - - - + + + - - - + + + - - - + + + - - - - + + - - - - + + - - + + + - - - + + + - - - + + + - - - + + + - - - - - - + + - - - - - + + + - - - - + + + - - - - - - - + + + + + - - - - - - - + + - - - - + + - - - - - - - - - + + + + - - - - + + - - - - - + + + - - - - + + + - - + + + + + + + - - + + + - + + - - - + + + - - + + + - - + + + - - - - - + + - - - + + - - - - - - - - - - + + + - - - - + + + - - + + - - - + + - - - - - - - - - - - - + + + - - - - - + + + - - - - - - + + - - - - - + + + - + + - - - + + - - - - - + + - - - - - - + + + - - - + + + - - - - + + + - - - - - + + + - - - - + + + - - + + + + + - - - + + - - + + - - - - - + + - - - - - - + + - - + - - - - + - - - - + - - - - + - - - - - + - + - - + - - - - + - - - - - + - - - - + - - - - - + - - + + + + + - - - + + + - - - + + + + + + + + + + + + + + + - + + + + + - - - - + + + + + - - - - - + + + + - - - + + + + + + + + - - - - + + + + + + + + - - - + + + + + + + - - - - + + + + + - - - - - + + + + + - - - + + + + - - - - - + + + + - - - - - - + + + + + - - - - - - + + + - - - - - - - - - - - - - - - - + + + - - - - + + + + + + - - - - - - + + + + + - - - - + + + + + + - - - - + + + + + + + + + + + - - - - - - - - + + + + + + + + + + + - - - - - - - + + + + + + + - - - - - - + + + + + + + - + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + - - + + + + + - - - - - - - - - - - - - + + + + + + + + + + + + + + - - - + + + + + + - - - - - - + + + + + - - - - - - + + + + + + + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - + + + + + + + + + + + - - - - - + + + + + - - - - - + + + + - - - - - - + + + + - - - - - - + + + + - - - - - - + + + + + - - - - - - + + - - - - - + + + + - - - - - + + - - - - - + + - - - - - - + + - - - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - + - - - - - - - - - + + - + + + + - + + + + - + - - - - - - - - - - - - - - - - + + - - + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + - - - - - - - - - + + + - - - - - - - - - - - - - - + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - + - + + + + + + + - - - - - + + + + + - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - + + + + - - - + + - - + + + + + + + + + + + + + - - + + + + + + + + + + + + + + + - - - + + + + + + + + + + + + + + + - - - - - - + + + + + - - - - - - + + + + - - - - - + + + + - - - - + + + - - - + + + + + - - - + + + + + - - - - + + + + - - - - + + + - - - - - + + + - - + + + - - + + + - - - + + + + + + - - - + + + + + - - - - - - + + + + - - - - - + + + + + - - + + + - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - + + - - - - - + + - - - - - + + - - - - - + + + + + + - - - + + + + - - + - - + - - + - - - - + - + - + - - - + + + + + + - - - - + + + - - - - - - - - - - - - - - - - - - - + + + + + + + + - - + + + + + + + - - - + + + + + - - - + + + + + - - - + + - - - + + - - + + + + + + - - - + + + + - - + + + + - - - - + + + + - - - - + + + + - - - - + + + + + - - - + + + - - + - - - + + + - - + - - - - + + - - - - + + - - - - - - - + + - - - - - + + + - - - - + + + - - - - + + + + - - - - - - - - - - - - - - - - - - - - + + + + - - - - + + + + - - - + + - - - + + - - - - + + - - - - - - - - + + - - - - - - + + - - - - - - - + + - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - + + - - + + - - + + - - + + - - + + - - + - - + - - + - - + - - - - - - - - + + - - - - - - - - - + + - - - - + - - - - + + - - - - + + - - - + - - + - - + - - - - - - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - + - - + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + - - - - + + + + - - - + + + - - - + + - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + - - - + + + - - - + + + + + + + + + + - + + - - + + - - + - - - + + + + + + + + + + + + - - + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + - - - + - - + - - + - - + + - - - + + - + - - + + - + - - + - - - - - + + - - + + - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - + + - - - + + - - - + + - - - + + - - - + + - + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + - + + + + + - + - - + + - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + + + + + + + + + - - + + - - - - + + - - - + + - - - + - - - + + - - - + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - + + + + - - - + + + - - - + + - - + + + + + + + + - - - - - - - - - + + + - - + + + + + - + + + + - - + + + + + - - - - - + + + + + - - + + + + + - - + + + + - - - + + + + - - - + + + + - - - + + + + - - + + + + - - + + + + + - - - + + + + - - - + + + - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - + + + + - - + + + + - - - + + + + + + + + - - - + + + + - - - - - + + + + - - + + + + - - + + + + + + + + - - - - + + - - + + + - - + - - + - - + - - - - - - - - - - - - - - - + + - - - + + - - + - - - + + - - - + + - - - + + - - - + + - - - + - - - + + - - + - - + - - - + + - - - + + - - - + + - - + + + + + + + + + + - - + - - + - - + + + - - + + - - - + + - - + - - + - - - + + - - - - + + + + - - - + + + - - - - - - - - - - - - - - - + - - - - - - + + - - - - - + + - - - - + + - - - - - - - - + + - - - - - - - - + + - - - - - - - + + - - - - - + + - - - - - + + - - - - + + - - - - + + - - - - - + + - - + - - + - - - - - - + + + + + - - - - - + + + - - - - - - + + + - - - - - - - - - - - + + + + - - - - - - - - - - - + + + + - - - - - - - + + + - - - - - - - + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + - - - - - - - + + + - - - - - - - + + - - - - - - - - - + + + + + - - - - - - + - - - - + + + + - - - - - + - + - - + - - + - - - + + - - - - + + - - - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - - - + + + + + + + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + - - - + + + + + - + + + + + - - - + + - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - + + - - - - + + + - - + + + - - - - - - - - - - - - - - + + + - - - - - - - - - - - - - - - - + + - - - - - - - - - - - - - - - + + - - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - - - + + - - - - - + + - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - - - - - + + - - - - - - + + - - - - - + + - - - + + - - - + + - - - - - + + - - - - - - + + - - - - + + - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - + + - - - - - - - - - + + - - - - + + - - + - - - - - + + - - - - - + + - - - - - + - - - + - - - + - - - + - - - + - - - - + - - + - - + + - - + - - + + - - + - - + - + - - - + + - - - - + + - - - - - + + - - - - + + - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - + - - + + - - - + + - - + + - - - + + - - - - + + - - + - - - + + - + - - - + + - - - - + + - - + - - + - - + + - - - - + + - - - - + + - - + - - - - - - - - - - - - + + - - + - - + + - + - - + + - - + + - - - - + + - + - - - - - - - + + - - - - - - + + - - - - + - - - + + - - - - - - - - - - - + + - - - - - + + - - - - - - - - - + + - - - - - - - - - + + - - - - - + + - - - - + + - - - - - - - - - - - - - - - - + + - - - - - - + + - - - - + + - - - - + + - - - - - - - - - - + + + + + + + + - - + - - - - + + - - - + + - - - + + - - - - - - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - - + + - - - + + - - - - + + - - - - + + - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - + + - - - - - + + - - - - - + + - - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + - - - - + + - - - - + + - - - - - - - - + + - - - - + + - - - - + + - - - - + + - - - - - - - - + + - + - - + + - - - + + - - - - - - - - - - - - - - + + - - - - + + - - + + - - - - - + + - - - + + - - + - - - - + + - - - - + + - - - + + - - - + + + + + + + + + + + + + + - - - + + - - - + + - - + - - + - - + + - - + + - - + + - - + + - - - - + - - - - + + - - - + + - - - - + + - - - - - - + + - - - - - - + + - - - + + - - - + + + + + - + - + - + - + @@ -13517,167 +17995,210 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - - - - - - - - + + - - - - + + - - + + - - - - + + - + - - + + - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -13689,16 +18210,95 @@ - - - - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -13709,16 +18309,136 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -13752,17 +18472,38 @@ + + + + + + + + + + + + + + + + + + + + + @@ -13770,19 +18511,97 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + @@ -13808,16 +18627,49 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index d24a7a6a4d73a..d3c906aa92701 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -857,12 +857,20 @@ PyAPI_FUNC(PyObject*) _PyUnicode_EncodeUTF16( /* --- Unicode-Escape Codecs ---------------------------------------------- */ +/* Variant of PyUnicode_DecodeUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); /* Helper for PyUnicode_DecodeUnicodeEscape that detects invalid escape chars. */ -PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscape( +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeInternal( const char *string, /* Unicode-Escape encoded string */ Py_ssize_t length, /* size of string */ const char *errors, /* error handling */ + Py_ssize_t *consumed, /* bytes consumed */ const char **first_invalid_escape /* on return, points to first invalid escaped char in string. */ diff --git a/Lib/encodings/unicode_escape.py b/Lib/encodings/unicode_escape.py index 817f93265a463..9b1ce99b339ae 100644 --- a/Lib/encodings/unicode_escape.py +++ b/Lib/encodings/unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 8d112a171d7c4..09ab852b39819 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2327,7 +2327,11 @@ def test_unicode_escape(self): (r"\x5c\x55\x30\x30\x31\x31\x30\x30\x30\x30", 10)) -class UnicodeEscapeTest(unittest.TestCase): +class UnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.unicode_escape_decode(b""), ("", 0)) @@ -2414,6 +2418,44 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '', + '', + '', + '\x00', + '\x00', + '\x00\t', + '\x00\t', + '\x00\t\n', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) class RawUnicodeEscapeTest(unittest.TestCase): def test_empty(self): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst new file mode 100644 index 0000000000000..c1c4ed1ace248 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-00-19-02.bpo-45461.4LB_tJ.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index 952072102d5d8..f22d4daca09fb 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -487,17 +487,20 @@ _codecs_utf_32_ex_decode_impl(PyObject *module, Py_buffer *data, _codecs.unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=3ca3c917176b82ab input=8328081a3a569bd6]*/ + const char *errors, int final) +/*[clinic end generated code: output=b284f97b12c635ee input=6154f039a9f7c639]*/ { - PyObject *decoded = PyUnicode_DecodeUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index 772c8ca538da2..4e2c057007b99 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1149,7 +1149,7 @@ _codecs_utf_32_ex_decode(PyObject *module, PyObject *const *args, Py_ssize_t nar } PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, -"unicode_escape_decode($module, data, errors=None, /)\n" +"unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1158,7 +1158,7 @@ PyDoc_STRVAR(_codecs_unicode_escape_decode__doc__, static PyObject * _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1166,8 +1166,9 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1208,8 +1209,20 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ _PyArg_BadArgument("unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + if (PyFloat_Check(args[2])) { + PyErr_SetString(PyExc_TypeError, + "integer argument expected, got float" ); + goto exit; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2922,4 +2935,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=51b42d170889524c input=a9049054013a1b77]*/ +/*[clinic end generated code: output=d4b696fe54cfee8f input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 38fb3ffc5eb37..d6fc03e1ae492 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6271,9 +6271,10 @@ PyUnicode_AsUTF16String(PyObject *unicode) static _PyUnicode_Name_CAPI *ucnhash_CAPI = NULL; PyObject * -_PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeInternal(const char *s, Py_ssize_t size, const char *errors, + Py_ssize_t *consumed, const char **first_invalid_escape) { const char *starts = s; @@ -6286,6 +6287,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, *first_invalid_escape = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } /* Escaped strings will always be longer than the resulting @@ -6336,7 +6340,7 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; - goto error; + goto incomplete; } c = (unsigned char) *s++; @@ -6390,7 +6394,10 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, count = 8; message = "truncated \\UXXXXXXXX escape"; hexescape: - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6403,12 +6410,9 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (count) { - goto error; - } /* when we get here, ch is a 32-bit unicode character */ if (ch > MAX_UNICODE) { @@ -6435,14 +6439,20 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } message = "malformed \\N character escape"; - if (s < end && *s == '{') { + if (s >= end) { + goto incomplete; + } + if (*s == '{') { const char *start = ++s; size_t namelen; /* look for the closing brace */ while (s < end && *s != '}') s++; + if (s >= end) { + goto incomplete; + } namelen = s - start; - if (namelen && s < end) { + if (namelen) { /* found a name. look it up in the unicode database */ s++; ch = 0xffffffff; /* in case 'getcode' messes up */ @@ -6468,6 +6478,11 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, continue; } + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } error: endinpos = s-starts; writer.min_length = end - s + writer.pos; @@ -6496,12 +6511,14 @@ _PyUnicode_DecodeUnicodeEscape(const char *s, } PyObject * -PyUnicode_DecodeUnicodeEscape(const char *s, +_PyUnicode_DecodeUnicodeEscapeStateful(const char *s, Py_ssize_t size, - const char *errors) + const char *errors, + Py_ssize_t *consumed) { const char *first_invalid_escape; - PyObject *result = _PyUnicode_DecodeUnicodeEscape(s, size, errors, + PyObject *result = _PyUnicode_DecodeUnicodeEscapeInternal(s, size, errors, + consumed, &first_invalid_escape); if (result == NULL) return NULL; @@ -6516,6 +6533,14 @@ PyUnicode_DecodeUnicodeEscape(const char *s, return result; } +PyObject * +PyUnicode_DecodeUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeUnicodeEscapeStateful(s, size, errors, NULL); +} + /* Return a Unicode-Escape string version of the Unicode object. */ PyObject * diff --git a/Parser/pegen/parse_string.c b/Parser/pegen/parse_string.c index 41fdc2d81d92b..f1df2c46a6cf6 100644 --- a/Parser/pegen/parse_string.c +++ b/Parser/pegen/parse_string.c @@ -120,7 +120,7 @@ decode_unicode_with_escapes(Parser *parser, const char *s, size_t len, Token *t) s = buf; const char *first_invalid_escape; - v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape); + v = _PyUnicode_DecodeUnicodeEscapeInternal(s, len, NULL, NULL, &first_invalid_escape); if (v != NULL && first_invalid_escape != NULL) { if (warn_invalid_escape_sequence(parser, *first_invalid_escape, t) < 0) { diff --git a/Python/ast.c b/Python/ast.c index c7ba4d9c544c8..6dd705926312f 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -4640,7 +4640,7 @@ decode_unicode_with_escapes(struct compiling *c, const node *n, const char *s, s = buf; const char *first_invalid_escape; - v = _PyUnicode_DecodeUnicodeEscape(s, len, NULL, &first_invalid_escape); + v = _PyUnicode_DecodeUnicodeEscapeInternal(s, len, NULL, NULL, &first_invalid_escape); if (v != NULL && first_invalid_escape != NULL) { if (warn_invalid_escape_sequence(c, n, *first_invalid_escape) < 0) { From webhook-mailer at python.org Thu Oct 14 13:04:24 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 17:04:24 -0000 Subject: [Python-checkins] bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) Message-ID: https://github.com/python/cpython/commit/39aa98346d5dd8ac591a7cafb467af21c53f1e5d commit: 39aa98346d5dd8ac591a7cafb467af21c53f1e5d branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-14T20:04:19+03:00 summary: bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.raw_unicode_escape_decode(). It is True by default to match the former behavior. files: A Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst M Include/cpython/unicodeobject.h M Lib/encodings/raw_unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index bc5a3b4bd0b99..ab4aebf5e70b9 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -796,6 +796,16 @@ PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeInternal( string. */ ); +/* --- Raw-Unicode-Escape Codecs ---------------------------------------------- */ + +/* Variant of PyUnicode_DecodeRawUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeRawUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); + /* --- Latin-1 Codecs ----------------------------------------------------- */ PyAPI_FUNC(PyObject*) _PyUnicode_AsLatin1String( diff --git a/Lib/encodings/raw_unicode_escape.py b/Lib/encodings/raw_unicode_escape.py index 2b919b40d3788..46c8e070dd192 100644 --- a/Lib/encodings/raw_unicode_escape.py +++ b/Lib/encodings/raw_unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.raw_unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.raw_unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.raw_unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.raw_unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 288a3006cdeb7..506b51c428fb5 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2483,7 +2483,11 @@ def test_partial(self): ] ) -class RawUnicodeEscapeTest(unittest.TestCase): +class RawUnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "raw-unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.raw_unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.raw_unicode_escape_decode(b""), ("", 0)) @@ -2532,6 +2536,35 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '\x00', + '\x00\t', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) + class EscapeEncodeTest(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst new file mode 100644 index 0000000000000..f2c0ae4ae51fb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "raw-unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index fc74127ce56c4..50afc097b3502 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -509,17 +509,20 @@ _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, _codecs.raw_unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=c98eeb56028070a6 input=d2f5159ce3b3392f]*/ + const char *errors, int final) +/*[clinic end generated code: output=11dbd96301e2879e input=2d166191beb3235a]*/ { - PyObject *decoded = PyUnicode_DecodeRawUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeRawUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index a7086dd6e18d7..855ac77a7f73f 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1143,7 +1143,7 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ } PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, -"raw_unicode_escape_decode($module, data, errors=None, /)\n" +"raw_unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1152,7 +1152,7 @@ PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1160,8 +1160,9 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1202,8 +1203,15 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss _PyArg_BadArgument("raw_unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2809,4 +2817,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=9e9fb1d5d81577e0 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=814dae36b6f885cb input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index af3b3335d60ca..386052f31bea2 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6379,8 +6379,6 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_ASCII_CHAR(ch) \ @@ -6407,7 +6405,7 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, continue; } - startinpos = s - starts - 1; + Py_ssize_t startinpos = s - starts - 1; /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; @@ -6554,8 +6552,8 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, *consumed = startinpos; break; } - error: - endinpos = s-starts; + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6735,9 +6733,10 @@ PyUnicode_AsUnicodeEscapeString(PyObject *unicode) /* --- Raw Unicode Escape Codec ------------------------------------------- */ PyObject * -PyUnicode_DecodeRawUnicodeEscape(const char *s, - Py_ssize_t size, - const char *errors) +_PyUnicode_DecodeRawUnicodeEscapeStateful(const char *s, + Py_ssize_t size, + const char *errors, + Py_ssize_t *consumed) { const char *starts = s; _PyUnicodeWriter writer; @@ -6746,6 +6745,9 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, PyObject *exc = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } @@ -6764,8 +6766,6 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_CHAR(ch) \ @@ -6780,11 +6780,21 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, } while(0) /* Non-escape characters are interpreted as Unicode ordinals */ - if (c != '\\' || s >= end) { + if (c != '\\' || (s >= end && !consumed)) { WRITE_CHAR(c); continue; } + Py_ssize_t startinpos = s - starts - 1; + /* \ - Escapes */ + if (s >= end) { + assert(consumed); + // Set message to silent compiler warning. + // Actually it is never used. + message = "\\ at end of string"; + goto incomplete; + } + c = (unsigned char) *s++; if (c == 'u') { count = 4; @@ -6800,10 +6810,12 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, WRITE_CHAR(c); continue; } - startinpos = s - starts - 2; /* \uHHHH with 4 hex digits, \U00HHHHHH with 8 */ - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6816,18 +6828,23 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (!count) { - if (ch <= MAX_UNICODE) { - WRITE_CHAR(ch); - continue; - } + if (ch > MAX_UNICODE) { message = "\\Uxxxxxxxx out of range"; + goto error; } + WRITE_CHAR(ch); + continue; - endinpos = s-starts; + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6849,7 +6866,14 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_XDECREF(errorHandler); Py_XDECREF(exc); return NULL; +} +PyObject * +PyUnicode_DecodeRawUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeRawUnicodeEscapeStateful(s, size, errors, NULL); } From webhook-mailer at python.org Thu Oct 14 14:23:51 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 18:23:51 -0000 Subject: [Python-checkins] [3.10] bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) (GH-28952) Message-ID: https://github.com/python/cpython/commit/4641afef661e6a22bc64194bd334b161c95edfe2 commit: 4641afef661e6a22bc64194bd334b161c95edfe2 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-14T21:23:39+03:00 summary: [3.10] bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) (GH-28952) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.raw_unicode_escape_decode(). It is True by default to match the former behavior. (cherry picked from commit 39aa98346d5dd8ac591a7cafb467af21c53f1e5d) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst M Doc/data/python3.10.abi M Include/cpython/unicodeobject.h M Lib/encodings/raw_unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c diff --git a/Doc/data/python3.10.abi b/Doc/data/python3.10.abi index 0e678f3aedf9f..94dcf74411b4b 100644 --- a/Doc/data/python3.10.abi +++ b/Doc/data/python3.10.abi @@ -1249,6 +1249,7 @@ + @@ -2357,7 +2358,7 @@ - + @@ -3376,10 +3377,10 @@ - + - + @@ -3991,7 +3992,7 @@ - + @@ -4648,7 +4649,7 @@ - + @@ -7573,7 +7574,7 @@ - + @@ -8021,7 +8022,7 @@ - + @@ -9066,7 +9067,7 @@ - + @@ -10097,7 +10098,7 @@ - + @@ -10480,7 +10481,7 @@ - + @@ -10892,351 +10893,358 @@ - + - + - - + + - - + + - - + + - - - + + + - - - - - + + + + + - - + + - - - - + + + + - - - - + + + + - - - + + + - - - + + + - - - + + + - - - - + + + + - - + + - - - - + + + + - - - - + + + + - - - - - + + + + + - - - - + + + + - - - - + + + + - - + + - - + + - + - - - + + + - - - + + + - + - - - + + + - - - + + + - - - + + + - - - + + + - - - - - + + + + + - - - - - + + + + + - - - - + + + + - - - + + + - - - - - - + + + + + + - - - - - - + + + + + + - - - - - - + + + + + + - - - - - + + + + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - - - + + + - - + + - - - - + + + + - - - - - + + + + + - - - + + + - - - - - + + + + + - - - - + + + + - - + + - - - - - + + + + + - + - - - - + + + + - - - - + + + + - + - - - - + + + + + + + + + + - - - - + + + - - - + + - - + + + + - - - - + + + + + - - - + + + - - + + - - - - + + + + - - - - - + + + + + @@ -11316,9 +11324,9 @@ - - - + + + @@ -11334,9 +11342,9 @@ - - - + + + @@ -11362,9 +11370,9 @@ - - - + + + @@ -11592,15 +11600,15 @@ - - - - + + + + - - - + + + @@ -11631,14 +11639,14 @@ - - - + + + - - - + + + @@ -11647,30 +11655,30 @@ - - - + + + - - - + + + - - - + + + - - - - - + + + + + - - + + @@ -11704,28 +11712,28 @@ - + - + - + - + - + - + - + @@ -11809,37 +11817,37 @@ - + - + - + - + - + - + - + - + - + - + - + @@ -12936,7 +12944,7 @@ - + @@ -18810,7 +18818,7 @@ - + @@ -20053,10 +20061,10 @@ - + - + @@ -20070,7 +20078,7 @@ - + @@ -20082,10 +20090,10 @@ - + - + @@ -20106,7 +20114,7 @@ - + diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index a00489534607d..0761f01efb42f 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -841,6 +841,7 @@ PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeStateful( const char *errors, /* error handling */ Py_ssize_t *consumed /* bytes consumed */ ); + /* Helper for PyUnicode_DecodeUnicodeEscape that detects invalid escape chars. */ PyAPI_FUNC(PyObject*) _PyUnicode_DecodeUnicodeEscapeInternal( @@ -865,6 +866,14 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_EncodeRawUnicodeEscape( Py_ssize_t length /* Number of Py_UNICODE chars to encode */ ); +/* Variant of PyUnicode_DecodeRawUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeRawUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); + /* --- Latin-1 Codecs ----------------------------------------------------- */ PyAPI_FUNC(PyObject*) _PyUnicode_AsLatin1String( diff --git a/Lib/encodings/raw_unicode_escape.py b/Lib/encodings/raw_unicode_escape.py index 2b919b40d3788..46c8e070dd192 100644 --- a/Lib/encodings/raw_unicode_escape.py +++ b/Lib/encodings/raw_unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.raw_unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.raw_unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.raw_unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.raw_unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 153912b241a09..f7310fbb1474a 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2471,7 +2471,11 @@ def test_partial(self): ] ) -class RawUnicodeEscapeTest(unittest.TestCase): +class RawUnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "raw-unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.raw_unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.raw_unicode_escape_decode(b""), ("", 0)) @@ -2520,6 +2524,35 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '\x00', + '\x00\t', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) + class EscapeEncodeTest(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst new file mode 100644 index 0000000000000..f2c0ae4ae51fb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "raw-unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index fc74127ce56c4..50afc097b3502 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -509,17 +509,20 @@ _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, _codecs.raw_unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=c98eeb56028070a6 input=d2f5159ce3b3392f]*/ + const char *errors, int final) +/*[clinic end generated code: output=11dbd96301e2879e input=2d166191beb3235a]*/ { - PyObject *decoded = PyUnicode_DecodeRawUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeRawUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index a7086dd6e18d7..855ac77a7f73f 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1143,7 +1143,7 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ } PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, -"raw_unicode_escape_decode($module, data, errors=None, /)\n" +"raw_unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1152,7 +1152,7 @@ PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1160,8 +1160,9 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1202,8 +1203,15 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss _PyArg_BadArgument("raw_unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2809,4 +2817,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=9e9fb1d5d81577e0 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=814dae36b6f885cb input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index e660834b4788f..c72871074b3eb 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6445,8 +6445,6 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_ASCII_CHAR(ch) \ @@ -6473,7 +6471,7 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, continue; } - startinpos = s - starts - 1; + Py_ssize_t startinpos = s - starts - 1; /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; @@ -6620,8 +6618,8 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, *consumed = startinpos; break; } - error: - endinpos = s-starts; + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6816,9 +6814,10 @@ PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, /* --- Raw Unicode Escape Codec ------------------------------------------- */ PyObject * -PyUnicode_DecodeRawUnicodeEscape(const char *s, - Py_ssize_t size, - const char *errors) +_PyUnicode_DecodeRawUnicodeEscapeStateful(const char *s, + Py_ssize_t size, + const char *errors, + Py_ssize_t *consumed) { const char *starts = s; _PyUnicodeWriter writer; @@ -6827,6 +6826,9 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, PyObject *exc = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } @@ -6845,8 +6847,6 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_CHAR(ch) \ @@ -6861,11 +6861,21 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, } while(0) /* Non-escape characters are interpreted as Unicode ordinals */ - if (c != '\\' || s >= end) { + if (c != '\\' || (s >= end && !consumed)) { WRITE_CHAR(c); continue; } + Py_ssize_t startinpos = s - starts - 1; + /* \ - Escapes */ + if (s >= end) { + assert(consumed); + // Set message to silent compiler warning. + // Actually it is never used. + message = "\\ at end of string"; + goto incomplete; + } + c = (unsigned char) *s++; if (c == 'u') { count = 4; @@ -6881,10 +6891,12 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, WRITE_CHAR(c); continue; } - startinpos = s - starts - 2; /* \uHHHH with 4 hex digits, \U00HHHHHH with 8 */ - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6897,18 +6909,23 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (!count) { - if (ch <= MAX_UNICODE) { - WRITE_CHAR(ch); - continue; - } + if (ch > MAX_UNICODE) { message = "\\Uxxxxxxxx out of range"; + goto error; } + WRITE_CHAR(ch); + continue; - endinpos = s-starts; + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6930,7 +6947,14 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_XDECREF(errorHandler); Py_XDECREF(exc); return NULL; +} +PyObject * +PyUnicode_DecodeRawUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeRawUnicodeEscapeStateful(s, size, errors, NULL); } From webhook-mailer at python.org Thu Oct 14 14:23:57 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 14 Oct 2021 18:23:57 -0000 Subject: [Python-checkins] bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) (GH-28953) Message-ID: https://github.com/python/cpython/commit/684860280687561f6312e206c4ccfbe4baa17e89 commit: 684860280687561f6312e206c4ccfbe4baa17e89 branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-14T21:23:52+03:00 summary: bpo-45467: Fix IncrementalDecoder and StreamReader in the "raw-unicode-escape" codec (GH-28944) (GH-28953) They support now splitting escape sequences between input chunks. Add the third parameter "final" in codecs.raw_unicode_escape_decode(). It is True by default to match the former behavior. (cherry picked from commit 39aa98346d5dd8ac591a7cafb467af21c53f1e5d) files: A Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst M Doc/data/python3.9.abi M Include/cpython/unicodeobject.h M Lib/encodings/raw_unicode_escape.py M Lib/test/test_codecs.py M Modules/_codecsmodule.c M Modules/clinic/_codecsmodule.c.h M Objects/unicodeobject.c diff --git a/Doc/data/python3.9.abi b/Doc/data/python3.9.abi index 3b202fa39b047..e2037436bda00 100644 --- a/Doc/data/python3.9.abi +++ b/Doc/data/python3.9.abi @@ -1290,6 +1290,7 @@ + @@ -4199,10 +4200,10 @@ - + - + @@ -4808,7 +4809,7 @@ - + @@ -5430,7 +5431,7 @@ - + @@ -7213,7 +7214,7 @@ - + @@ -7588,13 +7589,13 @@ - + - + @@ -7650,7 +7651,7 @@ - + @@ -8701,7 +8702,7 @@ - + @@ -9664,7 +9665,7 @@ - + @@ -10044,7 +10045,7 @@ - + @@ -10463,389 +10464,396 @@ - + - + - - + + - - - + + + - - - + + + - - - - + + + + - - - + + + - - - + + + - - - - + + + + - - - + + + - - + + - - + + - - + + - - + + - - - + + + - - - - - + + + + + - - + + - - - - + + + + - - - - + + + + - - - + + + - - - + + + - - - + + + - - - - + + + + - - + + - - - - + + + + - - - - + + + + - - - - - + + + + + - - - - + + + + - - - - + + + + - - + + - - + + - + - - - + + + - - - + + + - + - - - + + + - - - + + + - - - + + + - - - + + + - - - - - + + + + + - - - - - + + + + + - - - + + + - - - - - - + + + + + + - - - - - - + + + + + + - - - - - - + + + + + + - - - - - + + + + + - - - - - - - - - - + + + + + + + + + + - - - - - + + + + + - - - + + + - - + + - - - - + + + + - - - - - + + + + + - - - + + + - - - - - + + + + + - - - - + + + + - - + + - - - - - + + + + + - + - - - - + + + + - - - - + + + + - + - - - - + + + + - - - - + + + + - - - + + + - - + + - - - - + + + + - - - + + + + + - - + + + - - - - + + - - - - - + + + + + + + + + + + @@ -10925,9 +10933,9 @@ - - - + + + @@ -10968,9 +10976,9 @@ - - - + + + @@ -10996,9 +11004,9 @@ - - - + + + @@ -11223,15 +11231,15 @@ - - - - + + + + - - - + + + @@ -11258,14 +11266,14 @@ - - - + + + - - - + + + @@ -11274,36 +11282,36 @@ - - - - + + + + - - - + + + - - - + + + - - - + + + - - - - - + + + + + - - + + @@ -11340,25 +11348,25 @@ - + - + - + - + - + - + @@ -11442,37 +11450,37 @@ - + - + - + - + - + - + - + - + - + - + - + @@ -12271,7 +12279,7 @@ - + @@ -16829,7 +16837,7 @@ - + @@ -18033,10 +18041,10 @@ - + - + @@ -18047,7 +18055,7 @@ - + @@ -18056,10 +18064,10 @@ - + - + @@ -18080,7 +18088,7 @@ - + diff --git a/Include/cpython/unicodeobject.h b/Include/cpython/unicodeobject.h index d3c906aa92701..1b460c9f189ff 100644 --- a/Include/cpython/unicodeobject.h +++ b/Include/cpython/unicodeobject.h @@ -888,6 +888,14 @@ Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_EncodeRawUnicodeEscape( Py_ssize_t length /* Number of Py_UNICODE chars to encode */ ); +/* Variant of PyUnicode_DecodeRawUnicodeEscape that supports partial decoding. */ +PyAPI_FUNC(PyObject*) _PyUnicode_DecodeRawUnicodeEscapeStateful( + const char *string, /* Unicode-Escape encoded string */ + Py_ssize_t length, /* size of string */ + const char *errors, /* error handling */ + Py_ssize_t *consumed /* bytes consumed */ +); + /* --- Latin-1 Codecs ----------------------------------------------------- */ PyAPI_FUNC(PyObject*) _PyUnicode_AsLatin1String( diff --git a/Lib/encodings/raw_unicode_escape.py b/Lib/encodings/raw_unicode_escape.py index 2b919b40d3788..46c8e070dd192 100644 --- a/Lib/encodings/raw_unicode_escape.py +++ b/Lib/encodings/raw_unicode_escape.py @@ -21,15 +21,16 @@ class IncrementalEncoder(codecs.IncrementalEncoder): def encode(self, input, final=False): return codecs.raw_unicode_escape_encode(input, self.errors)[0] -class IncrementalDecoder(codecs.IncrementalDecoder): - def decode(self, input, final=False): - return codecs.raw_unicode_escape_decode(input, self.errors)[0] +class IncrementalDecoder(codecs.BufferedIncrementalDecoder): + def _buffer_decode(self, input, errors, final): + return codecs.raw_unicode_escape_decode(input, errors, final) class StreamWriter(Codec,codecs.StreamWriter): pass class StreamReader(Codec,codecs.StreamReader): - pass + def decode(self, input, errors='strict'): + return codecs.raw_unicode_escape_decode(input, errors, False) ### encodings module API diff --git a/Lib/test/test_codecs.py b/Lib/test/test_codecs.py index 09ab852b39819..2f7f93a29f662 100644 --- a/Lib/test/test_codecs.py +++ b/Lib/test/test_codecs.py @@ -2457,7 +2457,11 @@ def test_partial(self): ] ) -class RawUnicodeEscapeTest(unittest.TestCase): +class RawUnicodeEscapeTest(ReadTest, unittest.TestCase): + encoding = "raw-unicode-escape" + + test_lone_surrogates = None + def test_empty(self): self.assertEqual(codecs.raw_unicode_escape_encode(""), (b"", 0)) self.assertEqual(codecs.raw_unicode_escape_decode(b""), ("", 0)) @@ -2506,6 +2510,35 @@ def test_decode_errors(self): self.assertEqual(decode(br"\U00110000", "ignore"), ("", 10)) self.assertEqual(decode(br"\U00110000", "replace"), ("\ufffd", 10)) + def test_partial(self): + self.check_partial( + "\x00\t\n\r\\\xff\uffff\U00010000", + [ + '\x00', + '\x00\t', + '\x00\t\n', + '\x00\t\n\r', + '\x00\t\n\r', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff', + '\x00\t\n\r\\\xff\uffff\U00010000', + ] + ) + class EscapeEncodeTest(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst new file mode 100644 index 0000000000000..f2c0ae4ae51fb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-13-31-19.bpo-45467.Q7Ma6A.rst @@ -0,0 +1,2 @@ +Fix incremental decoder and stream reader in the "raw-unicode-escape" codec. +Previously they failed if the escape sequence was split. diff --git a/Modules/_codecsmodule.c b/Modules/_codecsmodule.c index f22d4daca09fb..cbe5cc50f1ca6 100644 --- a/Modules/_codecsmodule.c +++ b/Modules/_codecsmodule.c @@ -507,17 +507,20 @@ _codecs_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, _codecs.raw_unicode_escape_decode data: Py_buffer(accept={str, buffer}) errors: str(accept={str, NoneType}) = None + final: bool(accept={int}) = True / [clinic start generated code]*/ static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors) -/*[clinic end generated code: output=c98eeb56028070a6 input=d2f5159ce3b3392f]*/ + const char *errors, int final) +/*[clinic end generated code: output=11dbd96301e2879e input=2d166191beb3235a]*/ { - PyObject *decoded = PyUnicode_DecodeRawUnicodeEscape(data->buf, data->len, - errors); - return codec_tuple(decoded, data->len); + Py_ssize_t consumed = data->len; + PyObject *decoded = _PyUnicode_DecodeRawUnicodeEscapeStateful(data->buf, data->len, + errors, + final ? NULL : &consumed); + return codec_tuple(decoded, consumed); } /*[clinic input] diff --git a/Modules/clinic/_codecsmodule.c.h b/Modules/clinic/_codecsmodule.c.h index 4e2c057007b99..7ddc36de8a76c 100644 --- a/Modules/clinic/_codecsmodule.c.h +++ b/Modules/clinic/_codecsmodule.c.h @@ -1234,7 +1234,7 @@ _codecs_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_ } PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, -"raw_unicode_escape_decode($module, data, errors=None, /)\n" +"raw_unicode_escape_decode($module, data, errors=None, final=True, /)\n" "--\n" "\n"); @@ -1243,7 +1243,7 @@ PyDoc_STRVAR(_codecs_raw_unicode_escape_decode__doc__, static PyObject * _codecs_raw_unicode_escape_decode_impl(PyObject *module, Py_buffer *data, - const char *errors); + const char *errors, int final); static PyObject * _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ssize_t nargs) @@ -1251,8 +1251,9 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss PyObject *return_value = NULL; Py_buffer data = {NULL, NULL}; const char *errors = NULL; + int final = 1; - if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 2)) { + if (!_PyArg_CheckPositional("raw_unicode_escape_decode", nargs, 1, 3)) { goto exit; } if (PyUnicode_Check(args[0])) { @@ -1293,8 +1294,20 @@ _codecs_raw_unicode_escape_decode(PyObject *module, PyObject *const *args, Py_ss _PyArg_BadArgument("raw_unicode_escape_decode", "argument 2", "str or None", args[1]); goto exit; } + if (nargs < 3) { + goto skip_optional; + } + if (PyFloat_Check(args[2])) { + PyErr_SetString(PyExc_TypeError, + "integer argument expected, got float" ); + goto exit; + } + final = _PyLong_AsInt(args[2]); + if (final == -1 && PyErr_Occurred()) { + goto exit; + } skip_optional: - return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors); + return_value = _codecs_raw_unicode_escape_decode_impl(module, &data, errors, final); exit: /* Cleanup for data */ @@ -2935,4 +2948,4 @@ _codecs_lookup_error(PyObject *module, PyObject *arg) #ifndef _CODECS_CODE_PAGE_ENCODE_METHODDEF #define _CODECS_CODE_PAGE_ENCODE_METHODDEF #endif /* !defined(_CODECS_CODE_PAGE_ENCODE_METHODDEF) */ -/*[clinic end generated code: output=d4b696fe54cfee8f input=a9049054013a1b77]*/ +/*[clinic end generated code: output=eed7dc9312baf252 input=a9049054013a1b77]*/ diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index d6fc03e1ae492..7767d140e6c39 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -6308,8 +6308,6 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_ASCII_CHAR(ch) \ @@ -6336,7 +6334,7 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, continue; } - startinpos = s - starts - 1; + Py_ssize_t startinpos = s - starts - 1; /* \ - Escapes */ if (s >= end) { message = "\\ at end of string"; @@ -6483,8 +6481,8 @@ _PyUnicode_DecodeUnicodeEscapeInternal(const char *s, *consumed = startinpos; break; } - error: - endinpos = s-starts; + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6679,9 +6677,10 @@ PyUnicode_EncodeUnicodeEscape(const Py_UNICODE *s, /* --- Raw Unicode Escape Codec ------------------------------------------- */ PyObject * -PyUnicode_DecodeRawUnicodeEscape(const char *s, - Py_ssize_t size, - const char *errors) +_PyUnicode_DecodeRawUnicodeEscapeStateful(const char *s, + Py_ssize_t size, + const char *errors, + Py_ssize_t *consumed) { const char *starts = s; _PyUnicodeWriter writer; @@ -6690,6 +6689,9 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, PyObject *exc = NULL; if (size == 0) { + if (consumed) { + *consumed = 0; + } _Py_RETURN_UNICODE_EMPTY(); } @@ -6708,8 +6710,6 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, unsigned char c = (unsigned char) *s++; Py_UCS4 ch; int count; - Py_ssize_t startinpos; - Py_ssize_t endinpos; const char *message; #define WRITE_CHAR(ch) \ @@ -6724,11 +6724,21 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, } while(0) /* Non-escape characters are interpreted as Unicode ordinals */ - if (c != '\\' || s >= end) { + if (c != '\\' || (s >= end && !consumed)) { WRITE_CHAR(c); continue; } + Py_ssize_t startinpos = s - starts - 1; + /* \ - Escapes */ + if (s >= end) { + assert(consumed); + // Set message to silent compiler warning. + // Actually it is never used. + message = "\\ at end of string"; + goto incomplete; + } + c = (unsigned char) *s++; if (c == 'u') { count = 4; @@ -6744,10 +6754,12 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, WRITE_CHAR(c); continue; } - startinpos = s - starts - 2; /* \uHHHH with 4 hex digits, \U00HHHHHH with 8 */ - for (ch = 0; count && s < end; ++s, --count) { + for (ch = 0; count; ++s, --count) { + if (s >= end) { + goto incomplete; + } c = (unsigned char)*s; ch <<= 4; if (c >= '0' && c <= '9') { @@ -6760,18 +6772,23 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, ch += c - ('A' - 10); } else { - break; + goto error; } } - if (!count) { - if (ch <= MAX_UNICODE) { - WRITE_CHAR(ch); - continue; - } + if (ch > MAX_UNICODE) { message = "\\Uxxxxxxxx out of range"; + goto error; } + WRITE_CHAR(ch); + continue; - endinpos = s-starts; + incomplete: + if (consumed) { + *consumed = startinpos; + break; + } + error:; + Py_ssize_t endinpos = s-starts; writer.min_length = end - s + writer.pos; if (unicode_decode_call_errorhandler_writer( errors, &errorHandler, @@ -6793,7 +6810,14 @@ PyUnicode_DecodeRawUnicodeEscape(const char *s, Py_XDECREF(errorHandler); Py_XDECREF(exc); return NULL; +} +PyObject * +PyUnicode_DecodeRawUnicodeEscape(const char *s, + Py_ssize_t size, + const char *errors) +{ + return _PyUnicode_DecodeRawUnicodeEscapeStateful(s, size, errors, NULL); } From webhook-mailer at python.org Thu Oct 14 15:53:13 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 19:53:13 -0000 Subject: [Python-checkins] bpo-45439: Move _PyObject_VectorcallTstate() to pycore_call.h (GH-28893) Message-ID: https://github.com/python/cpython/commit/3cc56c828d2d8f8659ea49447234bf0d2b87cd64 commit: 3cc56c828d2d8f8659ea49447234bf0d2b87cd64 branch: main author: Victor Stinner committer: vstinner date: 2021-10-14T21:53:04+02:00 summary: bpo-45439: Move _PyObject_VectorcallTstate() to pycore_call.h (GH-28893) * Move _PyObject_VectorcallTstate() and _PyObject_FastCallTstate() to pycore_call.h (internal C API). * Convert PyObject_CallOneArg(), PyObject_Vectorcall(), _PyObject_FastCall() and PyVectorcall_Function() static inline functions to regular functions. * Add _PyVectorcall_FunctionInline() static inline function. * PyObject_Vectorcall(), _PyObject_FastCall(), and PyObject_CallOneArg() now call _PyThreadState_GET() rather than PyThreadState_Get(). files: M Include/cpython/abstract.h M Include/internal/pycore_call.h M Modules/_functoolsmodule.c M Objects/call.c M Objects/classobject.c M Python/context.c diff --git a/Include/cpython/abstract.h b/Include/cpython/abstract.h index 0814bfa62e1a4..55a742c31fada 100644 --- a/Include/cpython/abstract.h +++ b/Include/cpython/abstract.h @@ -58,72 +58,13 @@ PyVectorcall_NARGS(size_t n) return n & ~PY_VECTORCALL_ARGUMENTS_OFFSET; } -static inline vectorcallfunc -PyVectorcall_Function(PyObject *callable) -{ - PyTypeObject *tp; - Py_ssize_t offset; - vectorcallfunc ptr; - - assert(callable != NULL); - tp = Py_TYPE(callable); - if (!PyType_HasFeature(tp, Py_TPFLAGS_HAVE_VECTORCALL)) { - return NULL; - } - assert(PyCallable_Check(callable)); - - offset = tp->tp_vectorcall_offset; - assert(offset > 0); - memcpy(&ptr, (char *) callable + offset, sizeof(ptr)); - return ptr; -} - -/* Call the callable object 'callable' with the "vectorcall" calling - convention. - - args is a C array for positional arguments. - - nargsf is the number of positional arguments plus optionally the flag - PY_VECTORCALL_ARGUMENTS_OFFSET which means that the caller is allowed to - modify args[-1]. - - kwnames is a tuple of keyword names. The values of the keyword arguments - are stored in "args" after the positional arguments (note that the number - of keyword arguments does not change nargsf). kwnames can also be NULL if - there are no keyword arguments. +PyAPI_FUNC(vectorcallfunc) PyVectorcall_Function(PyObject *callable); - keywords must only contain strings and all keys must be unique. - - Return the result on success. Raise an exception and return NULL on - error. */ -static inline PyObject * -_PyObject_VectorcallTstate(PyThreadState *tstate, PyObject *callable, - PyObject *const *args, size_t nargsf, - PyObject *kwnames) -{ - vectorcallfunc func; - PyObject *res; - - assert(kwnames == NULL || PyTuple_Check(kwnames)); - assert(args != NULL || PyVectorcall_NARGS(nargsf) == 0); - - func = PyVectorcall_Function(callable); - if (func == NULL) { - Py_ssize_t nargs = PyVectorcall_NARGS(nargsf); - return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames); - } - res = func(callable, args, nargsf, kwnames); - return _Py_CheckFunctionResult(tstate, callable, res, NULL); -} - -static inline PyObject * -PyObject_Vectorcall(PyObject *callable, PyObject *const *args, - size_t nargsf, PyObject *kwnames) -{ - PyThreadState *tstate = PyThreadState_Get(); - return _PyObject_VectorcallTstate(tstate, callable, - args, nargsf, kwnames); -} +PyAPI_FUNC(PyObject *) PyObject_Vectorcall( + PyObject *callable, + PyObject *const *args, + size_t nargsf, + PyObject *kwnames); // Backwards compatibility aliases for API that was provisional in Python 3.8 #define _PyObject_Vectorcall PyObject_Vectorcall @@ -146,35 +87,13 @@ PyAPI_FUNC(PyObject *) PyObject_VectorcallDict( "tuple" and keyword arguments "dict". "dict" may also be NULL */ PyAPI_FUNC(PyObject *) PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *dict); -static inline PyObject * -_PyObject_FastCallTstate(PyThreadState *tstate, PyObject *func, PyObject *const *args, Py_ssize_t nargs) -{ - return _PyObject_VectorcallTstate(tstate, func, args, (size_t)nargs, NULL); -} - -/* Same as PyObject_Vectorcall except without keyword arguments */ -static inline PyObject * -_PyObject_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs) -{ - PyThreadState *tstate = PyThreadState_Get(); - return _PyObject_FastCallTstate(tstate, func, args, nargs); -} - -static inline PyObject * -PyObject_CallOneArg(PyObject *func, PyObject *arg) -{ - PyObject *_args[2]; - PyObject **args; - PyThreadState *tstate; - size_t nargsf; +// Same as PyObject_Vectorcall(), except without keyword arguments +PyAPI_FUNC(PyObject *) _PyObject_FastCall( + PyObject *func, + PyObject *const *args, + Py_ssize_t nargs); - assert(arg != NULL); - args = _args + 1; // For PY_VECTORCALL_ARGUMENTS_OFFSET - args[0] = arg; - tstate = PyThreadState_Get(); - nargsf = 1 | PY_VECTORCALL_ARGUMENTS_OFFSET; - return _PyObject_VectorcallTstate(tstate, func, args, nargsf, NULL); -} +PyAPI_FUNC(PyObject *) PyObject_CallOneArg(PyObject *func, PyObject *arg); PyAPI_FUNC(PyObject *) PyObject_VectorcallMethod( PyObject *name, PyObject *const *args, diff --git a/Include/internal/pycore_call.h b/Include/internal/pycore_call.h index 933d714f13377..f2cfd2fd53ed9 100644 --- a/Include/internal/pycore_call.h +++ b/Include/internal/pycore_call.h @@ -30,11 +30,73 @@ PyAPI_FUNC(PyObject *) _PyObject_Call( PyObject *args, PyObject *kwargs); + +// Static inline variant of public PyVectorcall_Function(). +static inline vectorcallfunc +_PyVectorcall_FunctionInline(PyObject *callable) +{ + assert(callable != NULL); + + PyTypeObject *tp = Py_TYPE(callable); + if (!PyType_HasFeature(tp, Py_TPFLAGS_HAVE_VECTORCALL)) { + return NULL; + } + assert(PyCallable_Check(callable)); + + Py_ssize_t offset = tp->tp_vectorcall_offset; + assert(offset > 0); + + vectorcallfunc ptr; + memcpy(&ptr, (char *) callable + offset, sizeof(ptr)); + return ptr; +} + + +/* Call the callable object 'callable' with the "vectorcall" calling + convention. + + args is a C array for positional arguments. + + nargsf is the number of positional arguments plus optionally the flag + PY_VECTORCALL_ARGUMENTS_OFFSET which means that the caller is allowed to + modify args[-1]. + + kwnames is a tuple of keyword names. The values of the keyword arguments + are stored in "args" after the positional arguments (note that the number + of keyword arguments does not change nargsf). kwnames can also be NULL if + there are no keyword arguments. + + keywords must only contain strings and all keys must be unique. + + Return the result on success. Raise an exception and return NULL on + error. */ +static inline PyObject * +_PyObject_VectorcallTstate(PyThreadState *tstate, PyObject *callable, + PyObject *const *args, size_t nargsf, + PyObject *kwnames) +{ + vectorcallfunc func; + PyObject *res; + + assert(kwnames == NULL || PyTuple_Check(kwnames)); + assert(args != NULL || PyVectorcall_NARGS(nargsf) == 0); + + func = _PyVectorcall_FunctionInline(callable); + if (func == NULL) { + Py_ssize_t nargs = PyVectorcall_NARGS(nargsf); + return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwnames); + } + res = func(callable, args, nargsf, kwnames); + return _Py_CheckFunctionResult(tstate, callable, res, NULL); +} + + static inline PyObject * _PyObject_CallNoArgsTstate(PyThreadState *tstate, PyObject *func) { return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } + // Private static inline function variant of public PyObject_CallNoArgs() static inline PyObject * _PyObject_CallNoArgs(PyObject *func) { @@ -42,6 +104,14 @@ _PyObject_CallNoArgs(PyObject *func) { return _PyObject_VectorcallTstate(tstate, func, NULL, 0, NULL); } + +static inline PyObject * +_PyObject_FastCallTstate(PyThreadState *tstate, PyObject *func, PyObject *const *args, Py_ssize_t nargs) +{ + return _PyObject_VectorcallTstate(tstate, func, args, (size_t)nargs, NULL); +} + + #ifdef __cplusplus } #endif diff --git a/Modules/_functoolsmodule.c b/Modules/_functoolsmodule.c index 4c77ee7b7a53c..1f6b852f6d99b 100644 --- a/Modules/_functoolsmodule.c +++ b/Modules/_functoolsmodule.c @@ -269,7 +269,7 @@ partial_vectorcall(partialobject *pto, PyObject *const *args, static void partial_setvectorcall(partialobject *pto) { - if (PyVectorcall_Function(pto->fn) == NULL) { + if (_PyVectorcall_Function(pto->fn) == NULL) { /* Don't use vectorcall if the underlying function doesn't support it */ pto->vectorcall = NULL; } diff --git a/Objects/call.c b/Objects/call.c index cfcd4220f3cf7..5e55518b04cbf 100644 --- a/Objects/call.c +++ b/Objects/call.c @@ -109,8 +109,7 @@ _Py_CheckSlotResult(PyObject *obj, const char *slot_name, int success) PyObject * PyObject_CallNoArgs(PyObject *func) { - PyThreadState *tstate = _PyThreadState_GET(); - return _PyObject_CallNoArgsTstate(tstate, func); + return _PyObject_CallNoArgs(func); } @@ -131,7 +130,7 @@ _PyObject_FastCallDictTstate(PyThreadState *tstate, PyObject *callable, assert(nargs == 0 || args != NULL); assert(kwargs == NULL || PyDict_Check(kwargs)); - vectorcallfunc func = PyVectorcall_Function(callable); + vectorcallfunc func = _PyVectorcall_Function(callable); if (func == NULL) { /* Use tp_call instead */ return _PyObject_MakeTpCall(tstate, callable, args, nargs, kwargs); @@ -225,6 +224,13 @@ _PyObject_MakeTpCall(PyThreadState *tstate, PyObject *callable, } +vectorcallfunc +PyVectorcall_Function(PyObject *callable) +{ + return _PyVectorcall_FunctionInline(callable); +} + + static PyObject * _PyVectorcall_Call(PyThreadState *tstate, vectorcallfunc func, PyObject *callable, PyObject *tuple, PyObject *kwargs) @@ -260,7 +266,7 @@ PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) { PyThreadState *tstate = _PyThreadState_GET(); - /* get vectorcallfunc as in PyVectorcall_Function, but without + /* get vectorcallfunc as in _PyVectorcall_Function, but without * the Py_TPFLAGS_HAVE_VECTORCALL check */ Py_ssize_t offset = Py_TYPE(callable)->tp_vectorcall_offset; if (offset <= 0) { @@ -284,6 +290,24 @@ PyVectorcall_Call(PyObject *callable, PyObject *tuple, PyObject *kwargs) } +PyObject * +PyObject_Vectorcall(PyObject *callable, PyObject *const *args, + size_t nargsf, PyObject *kwnames) +{ + PyThreadState *tstate = _PyThreadState_GET(); + return _PyObject_VectorcallTstate(tstate, callable, + args, nargsf, kwnames); +} + + +PyObject * +_PyObject_FastCall(PyObject *func, PyObject *const *args, Py_ssize_t nargs) +{ + PyThreadState *tstate = _PyThreadState_GET(); + return _PyObject_FastCallTstate(tstate, func, args, nargs); +} + + PyObject * _PyObject_Call(PyThreadState *tstate, PyObject *callable, PyObject *args, PyObject *kwargs) @@ -298,7 +322,7 @@ _PyObject_Call(PyThreadState *tstate, PyObject *callable, assert(PyTuple_Check(args)); assert(kwargs == NULL || PyDict_Check(kwargs)); - vectorcallfunc vector_func = PyVectorcall_Function(callable); + vectorcallfunc vector_func = _PyVectorcall_Function(callable); if (vector_func != NULL) { return _PyVectorcall_Call(tstate, vector_func, callable, args, kwargs); } @@ -339,6 +363,19 @@ PyCFunction_Call(PyObject *callable, PyObject *args, PyObject *kwargs) } +PyObject * +PyObject_CallOneArg(PyObject *func, PyObject *arg) +{ + assert(arg != NULL); + PyObject *_args[2]; + PyObject **args = _args + 1; // For PY_VECTORCALL_ARGUMENTS_OFFSET + args[0] = arg; + PyThreadState *tstate = _PyThreadState_GET(); + size_t nargsf = 1 | PY_VECTORCALL_ARGUMENTS_OFFSET; + return _PyObject_VectorcallTstate(tstate, func, args, nargsf, NULL); +} + + /* --- PyFunction call functions ---------------------------------- */ PyObject * diff --git a/Objects/classobject.c b/Objects/classobject.c index af73be3d26241..9d4fc99f1f858 100644 --- a/Objects/classobject.c +++ b/Objects/classobject.c @@ -1,6 +1,7 @@ /* Class object implementation (dead now except for methods) */ #include "Python.h" +#include "pycore_call.h" // _PyObject_VectorcallTstate() #include "pycore_object.h" #include "pycore_pyerrors.h" #include "pycore_pystate.h" // _PyThreadState_GET() diff --git a/Python/context.c b/Python/context.c index ad47992d9e3cd..d78f7f993bb89 100644 --- a/Python/context.c +++ b/Python/context.c @@ -1,5 +1,5 @@ #include "Python.h" - +#include "pycore_call.h" // _PyObject_VectorcallTstate() #include "pycore_context.h" #include "pycore_gc.h" // _PyObject_GC_MAY_BE_TRACKED() #include "pycore_hamt.h" From webhook-mailer at python.org Thu Oct 14 16:48:45 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Thu, 14 Oct 2021 20:48:45 -0000 Subject: [Python-checkins] bpo-45471: Do not set PyConfig.stdlib_dir in Py_SetPythonHome(). (gh-28954) Message-ID: https://github.com/python/cpython/commit/0bbea0723ee07f9d7ad9745f0e1875718ef38715 commit: 0bbea0723ee07f9d7ad9745f0e1875718ef38715 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-14T14:48:32-06:00 summary: bpo-45471: Do not set PyConfig.stdlib_dir in Py_SetPythonHome(). (gh-28954) The change in gh-28586 (bpo-45211) should not have included code to set _Py_path_config.stdlib_dir in Py_SetPythonHome(). We fix that here. https://bugs.python.org/issue45471 files: M Lib/test/test_embed.py M Python/pathconfig.c diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 41e092019c49a..4cbb4c2c1ce36 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1247,6 +1247,11 @@ def test_init_setpythonhome(self): self.fail(f"Unable to find home in {paths!r}") prefix = exec_prefix = home + if MS_WINDOWS: + stdlib = os.path.join(home, sys.platlibdir) + else: + version = f'{sys.version_info.major}.{sys.version_info.minor}' + stdlib = os.path.join(home, sys.platlibdir, f'python{version}') expected_paths = self.module_search_paths(prefix=home, exec_prefix=home) config = { @@ -1257,7 +1262,7 @@ def test_init_setpythonhome(self): 'exec_prefix': exec_prefix, 'base_exec_prefix': exec_prefix, 'pythonpath_env': paths_str, - 'stdlib_dir': home, + 'stdlib_dir': stdlib, } self.default_program_name(config) env = {'TESTHOME': home, 'PYTHONPATH': paths_str} diff --git a/Python/pathconfig.c b/Python/pathconfig.c index 2ebb9d39b8c2c..ad22222e000fc 100644 --- a/Python/pathconfig.c +++ b/Python/pathconfig.c @@ -530,13 +530,10 @@ Py_SetPythonHome(const wchar_t *home) PyMem_RawFree(_Py_path_config.home); _Py_path_config.home = _PyMem_RawWcsdup(home); - if (_Py_path_config.home != NULL) { - _Py_path_config.stdlib_dir = _PyMem_RawWcsdup(home); - } PyMem_SetAllocator(PYMEM_DOMAIN_RAW, &old_alloc); - if (_Py_path_config.home == NULL || _Py_path_config.stdlib_dir == NULL) { + if (_Py_path_config.home == NULL) { path_out_of_memory(__func__); } } From webhook-mailer at python.org Thu Oct 14 16:59:55 2021 From: webhook-mailer at python.org (ethanfurman) Date: Thu, 14 Oct 2021 20:59:55 -0000 Subject: [Python-checkins] bpo-45417: [Enum] fix quadratic behavior during creation (GH-28907) Message-ID: https://github.com/python/cpython/commit/b2af211e229df941d9b404f69547a264115156b5 commit: b2af211e229df941d9b404f69547a264115156b5 branch: main author: Carl Friedrich Bolz-Tereick committer: ethanfurman date: 2021-10-14T13:59:51-07:00 summary: bpo-45417: [Enum] fix quadratic behavior during creation (GH-28907) Creating an Enum exhibited quadratic behavior based on the number of members in three places: - `EnumDict._member_names`: a list searched with each new member's name - member creation: a `for` loop checking each existing member to see if new member was a duplicate - `auto()` values: a list of all previous values in enum was copied before being sent to `_generate_next_value()` Two of those issues have been resolved: - `_EnumDict._member_names` is now a dictionary so lookups are fast - member creation tries a fast value lookup before falling back to the slower `for` loop lookup The third issue still remains, as `_generate_next_value_()` can be user-overridden and could corrupt the last values list if it were not copied. files: A Misc/NEWS.d/next/Library/2021-10-12-20-35-06.bpo-45417.gQM-O7.rst M Lib/enum.py diff --git a/Lib/enum.py b/Lib/enum.py index 0776761ae6e73..461d276eed862 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -235,11 +235,18 @@ def __set_name__(self, enum_class, member_name): enum_member._sort_order_ = len(enum_class._member_names_) # If another member with the same value was already defined, the # new member becomes an alias to the existing one. - for name, canonical_member in enum_class._member_map_.items(): - if canonical_member._value_ == enum_member._value_: - enum_member = canonical_member - break - else: + try: + try: + # try to do a fast lookup to avoid the quadratic loop + enum_member = enum_class._value2member_map_[value] + except TypeError: + for name, canonical_member in enum_class._member_map_.items(): + if canonical_member._value_ == value: + enum_member = canonical_member + break + else: + raise KeyError + except KeyError: # this could still be an alias if the value is multi-bit and the # class is a flag class if ( @@ -301,7 +308,7 @@ class _EnumDict(dict): """ def __init__(self): super().__init__() - self._member_names = [] + self._member_names = {} # use a dict to keep insertion order self._last_values = [] self._ignore = [] self._auto_called = False @@ -365,7 +372,7 @@ def __setitem__(self, key, value): ) self._auto_called = True value = value.value - self._member_names.append(key) + self._member_names[key] = None self._last_values.append(value) super().__setitem__(key, value) diff --git a/Misc/NEWS.d/next/Library/2021-10-12-20-35-06.bpo-45417.gQM-O7.rst b/Misc/NEWS.d/next/Library/2021-10-12-20-35-06.bpo-45417.gQM-O7.rst new file mode 100644 index 0000000000000..a15c23917a7e6 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-12-20-35-06.bpo-45417.gQM-O7.rst @@ -0,0 +1,2 @@ +Fix quadratic behaviour in the enum module: Creation of enum classes with a +lot of entries was quadratic. From webhook-mailer at python.org Thu Oct 14 17:32:30 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Thu, 14 Oct 2021 21:32:30 -0000 Subject: [Python-checkins] bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656) Message-ID: https://github.com/python/cpython/commit/79cf20e48d0b5d69d9fac2a0204b5ac2c366066a commit: 79cf20e48d0b5d69d9fac2a0204b5ac2c366066a branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-14T15:32:18-06:00 summary: bpo-21736: Set __file__ on frozen stdlib modules. (gh-28656) Currently frozen modules do not have __file__ set. In their spec, origin is set to "frozen" and they are marked as not having a location. (Similarly, for frozen packages __path__ is set to an empty list.) However, for frozen stdlib modules we are able to extrapolate __file__ as long as we can determine the stdlib directory at runtime. (We now do so since gh-28586.) Having __file__ set is helpful for a number of reasons. Likewise, having a non-empty __path__ means we can import submodules of a frozen package from the filesystem (e.g. we could partially freeze the encodings module). This change sets __file__ (and adds to __path__) for frozen stdlib modules. It uses sys._stdlibdir (from gh-28586) and the frozen module alias information (from gh-28655). All that work is done in FrozenImporter (in Lib/importlib/_bootstrap.py). Also, if a frozen module is imported before importlib is bootstrapped (during interpreter initialization) then we fix up that module and its spec during the importlib bootstrapping step (i.e. imporlib._bootstrap._setup()) to match what gets set by FrozenImporter, including setting the file info (if the stdlib dir is known). To facilitate this, modules imported using PyImport_ImportFrozenModule() have __origname__ set using the frozen module alias info. __origname__ is popped off during importlib bootstrap. (To be clear, even with this change the new code to set __file__ during fixups in imporlib._bootstrap._setup() doesn't actually get triggered yet. This is because sys._stdlibdir hasn't been set yet in interpreter initialization at the point importlib is bootstrapped. However, we do fix up such modules at that point to otherwise match the result of importing through FrozenImporter, just not the __file__ and __path__ parts. Doing so will require changes in the order in which things happen during interpreter initialization. That can be addressed separately. Once it is, the file-related fixup code from this PR will kick in.) Here are things this change does not do: * set __file__ for non-stdlib modules (no way of knowing the parent dir) * set __file__ if the stdlib dir is not known (nor assume the expense of finding it) * relatedly, set __file__ if the stdlib is in a zip file * verify that the filename set to __file__ actually exists (too expensive) * update __path__ for frozen packages that alias a non-package (since there is no package dir) Other things this change skips, but we may do later: * set __file__ on modules imported using PyImport_ImportFrozenModule() * set co_filename when we unmarshal the frozen code object while importing the module (e.g. in FrozenImporter.exec_module()) -- this would allow tracebacks to show source lines * implement FrozenImporter.get_filename() and FrozenImporter.get_source() https://bugs.python.org/issue21736 files: A Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-21-02.bpo-21736.RI47BU.rst M Lib/importlib/_bootstrap.py M Lib/test/test_frozen.py M Lib/test/test_importlib/frozen/test_finder.py M Lib/test/test_importlib/frozen/test_loader.py M Python/clinic/import.c.h M Python/import.c diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index 49f08814e9518..889f08f8aeec1 100644 --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -421,7 +421,10 @@ def has_location(self, value): def spec_from_loader(name, loader, *, origin=None, is_package=None): """Return a module spec based on various loader methods.""" - if hasattr(loader, 'get_filename'): + if origin is None: + origin = getattr(loader, '_ORIGIN', None) + + if not origin and hasattr(loader, 'get_filename'): if _bootstrap_external is None: raise NotImplementedError spec_from_file_location = _bootstrap_external.spec_from_file_location @@ -467,12 +470,9 @@ def _spec_from_module(module, loader=None, origin=None): except AttributeError: location = None if origin is None: - if location is None: - try: - origin = loader._ORIGIN - except AttributeError: - origin = None - else: + if loader is not None: + origin = getattr(loader, '_ORIGIN', None) + if not origin and location is not None: origin = location try: cached = module.__cached__ @@ -484,7 +484,7 @@ def _spec_from_module(module, loader=None, origin=None): submodule_search_locations = None spec = ModuleSpec(name, loader, origin=origin) - spec._set_fileattr = False if location is None else True + spec._set_fileattr = False if location is None else (origin == location) spec.cached = cached spec.submodule_search_locations = submodule_search_locations return spec @@ -541,6 +541,7 @@ def _init_module_attrs(spec, module, *, override=False): # __path__ if override or getattr(module, '__path__', None) is None: if spec.submodule_search_locations is not None: + # XXX We should extend __path__ if it's already a list. try: module.__path__ = spec.submodule_search_locations except AttributeError: @@ -825,38 +826,127 @@ def module_repr(m): return ''.format(m.__name__, FrozenImporter._ORIGIN) @classmethod - def _setup_module(cls, module): - assert not hasattr(module, '__file__'), module.__file__ - ispkg = hasattr(module, '__path__') - assert not ispkg or not module.__path__, module.__path__ + def _fix_up_module(cls, module): spec = module.__spec__ - assert not ispkg or not spec.submodule_search_locations + state = spec.loader_state + if state is None: + # The module is missing FrozenImporter-specific values. - if spec.loader_state is None: - spec.loader_state = type(sys.implementation)( - data=None, - origname=None, - ) - elif not hasattr(spec.loader_state, 'data'): - spec.loader_state.data = None - if not getattr(spec.loader_state, 'origname', None): + # Fix up the spec attrs. origname = vars(module).pop('__origname__', None) assert origname, 'see PyImport_ImportFrozenModuleObject()' - spec.loader_state.origname = origname + ispkg = hasattr(module, '__path__') + assert _imp.is_frozen_package(module.__name__) == ispkg, ispkg + filename, pkgdir = cls._resolve_filename(origname, spec.name, ispkg) + spec.loader_state = type(sys.implementation)( + filename=filename, + origname=origname, + ) + __path__ = spec.submodule_search_locations + if ispkg: + assert __path__ == [], __path__ + if pkgdir: + spec.submodule_search_locations.insert(0, pkgdir) + else: + assert __path__ is None, __path__ + + # Fix up the module attrs (the bare minimum). + assert not hasattr(module, '__file__'), module.__file__ + if filename: + try: + module.__file__ = filename + except AttributeError: + pass + if ispkg: + if module.__path__ != __path__: + assert module.__path__ == [], module.__path__ + module.__path__.extend(__path__) + else: + # These checks ensure that _fix_up_module() is only called + # in the right places. + __path__ = spec.submodule_search_locations + ispkg = __path__ is not None + # Check the loader state. + assert sorted(vars(state)) == ['filename', 'origname'], state + if state.origname: + # The only frozen modules with "origname" set are stdlib modules. + (__file__, pkgdir, + ) = cls._resolve_filename(state.origname, spec.name, ispkg) + assert state.filename == __file__, (state.filename, __file__) + if pkgdir: + assert __path__ == [pkgdir], (__path__, pkgdir) + else: + assert __path__ == ([] if ispkg else None), __path__ + else: + __file__ = None + assert state.filename is None, state.filename + assert __path__ == ([] if ispkg else None), __path__ + # Check the file attrs. + if __file__: + assert hasattr(module, '__file__') + assert module.__file__ == __file__, (module.__file__, __file__) + else: + assert not hasattr(module, '__file__'), module.__file__ + if ispkg: + assert hasattr(module, '__path__') + assert module.__path__ == __path__, (module.__path__, __path__) + else: + assert not hasattr(module, '__path__'), module.__path__ + assert not spec.has_location + + @classmethod + def _resolve_filename(cls, fullname, alias=None, ispkg=False): + if not fullname or not getattr(sys, '_stdlib_dir', None): + return None, None + try: + sep = cls._SEP + except AttributeError: + sep = cls._SEP = '\\' if sys.platform == 'win32' else '/' + + if fullname != alias: + if fullname.startswith('<'): + fullname = fullname[1:] + if not ispkg: + fullname = f'{fullname}.__init__' + else: + ispkg = False + relfile = fullname.replace('.', sep) + if ispkg: + pkgdir = f'{sys._stdlib_dir}{sep}{relfile}' + filename = f'{pkgdir}{sep}__init__.py' + else: + pkgdir = None + filename = f'{sys._stdlib_dir}{sep}{relfile}.py' + return filename, pkgdir @classmethod def find_spec(cls, fullname, path=None, target=None): info = _call_with_frames_removed(_imp.find_frozen, fullname) if info is None: return None - data, ispkg, origname = info + # We get the marshaled data in exec_module() (the loader + # part of the importer), instead of here (the finder part). + # The loader is the usual place to get the data that will + # be loaded into the module. (For example, see _LoaderBasics + # in _bootstra_external.py.) Most importantly, this importer + # is simpler if we wait to get the data. + # However, getting as much data in the finder as possible + # to later load the module is okay, and sometimes important. + # (That's why ModuleSpec.loader_state exists.) This is + # especially true if it avoids throwing away expensive data + # the loader would otherwise duplicate later and can be done + # efficiently. In this case it isn't worth it. + _, ispkg, origname = info spec = spec_from_loader(fullname, cls, origin=cls._ORIGIN, is_package=ispkg) + filename, pkgdir = cls._resolve_filename(origname, fullname, ispkg) spec.loader_state = type(sys.implementation)( - data=data, + filename=filename, origname=origname, ) + if pkgdir: + spec.submodule_search_locations.insert(0, pkgdir) return spec @classmethod @@ -873,26 +963,22 @@ def find_module(cls, fullname, path=None): @staticmethod def create_module(spec): - """Use default semantics for module creation.""" + """Set __file__, if able.""" + module = _new_module(spec.name) + try: + filename = spec.loader_state.filename + except AttributeError: + pass + else: + if filename: + module.__file__ = filename + return module @staticmethod def exec_module(module): spec = module.__spec__ name = spec.name - try: - data = spec.loader_state.data - except AttributeError: - if not _imp.is_frozen(name): - raise ImportError('{!r} is not a frozen module'.format(name), - name=name) - data = None - else: - # We clear the extra data we got from the finder, to save memory. - # Note that if this method is called again (e.g. by - # importlib.reload()) then _imp.get_frozen_object() will notice - # no data was provided and will look it up. - spec.loader_state.data = None - code = _call_with_frames_removed(_imp.get_frozen_object, name, data) + code = _call_with_frames_removed(_imp.get_frozen_object, name) exec(code, module.__dict__) @classmethod @@ -903,7 +989,16 @@ def load_module(cls, fullname): """ # Warning about deprecation implemented in _load_module_shim(). - return _load_module_shim(cls, fullname) + module = _load_module_shim(cls, fullname) + info = _imp.find_frozen(fullname) + assert info is not None + _, ispkg, origname = info + module.__origname__ = origname + vars(module).pop('__file__', None) + if ispkg: + module.__path__ = [] + cls._fix_up_module(module) + return module @classmethod @_requires_frozen @@ -1244,7 +1339,7 @@ def _setup(sys_module, _imp_module): spec = _spec_from_module(module, loader) _init_module_attrs(spec, module) if loader is FrozenImporter: - loader._setup_module(module) + loader._fix_up_module(module) # Directly load built-in modules needed during bootstrap. self_module = sys.modules[__name__] diff --git a/Lib/test/test_frozen.py b/Lib/test/test_frozen.py index 029fd068793c3..0b4a12bcf4094 100644 --- a/Lib/test/test_frozen.py +++ b/Lib/test/test_frozen.py @@ -39,9 +39,6 @@ def test_frozen_submodule_in_unfrozen_package(self): self.assertIs(spam.__spec__.loader, importlib.machinery.FrozenImporter) - # This is not possible until frozen packages have __path__ set properly. - # See https://bugs.python.org/issue21736. - @unittest.expectedFailure def test_unfrozen_submodule_in_frozen_package(self): with import_helper.CleanImport('__phello__', '__phello__.spam'): with import_helper.frozen_modules(enabled=True): diff --git a/Lib/test/test_importlib/frozen/test_finder.py b/Lib/test/test_importlib/frozen/test_finder.py index cd5586d524ce2..66080b2ade009 100644 --- a/Lib/test/test_importlib/frozen/test_finder.py +++ b/Lib/test/test_importlib/frozen/test_finder.py @@ -44,30 +44,31 @@ def check_loader_state(self, spec, origname=None, filename=None): if not filename: if not origname: origname = spec.name + filename = resolve_stdlib_file(origname) actual = dict(vars(spec.loader_state)) - # Check the code object used to import the frozen module. - # We can't compare the marshaled data directly because - # marshal.dumps() would mark "expected" (below) as a ref, - # which slightly changes the output. - # (See https://bugs.python.org/issue34093.) - data = actual.pop('data') - with import_helper.frozen_modules(): - expected = _imp.get_frozen_object(spec.name) - code = marshal.loads(data) - self.assertEqual(code, expected) - # Check the rest of spec.loader_state. expected = dict( origname=origname, + filename=filename if origname else None, ) self.assertDictEqual(actual, expected) def check_search_locations(self, spec): - # Frozen packages do not have any path entries. - # (See https://bugs.python.org/issue21736.) - expected = [] + """This is only called when testing packages.""" + missing = object() + filename = getattr(spec.loader_state, 'filename', missing) + origname = getattr(spec.loader_state, 'origname', None) + if not origname or filename is missing: + # We deal with this in check_loader_state(). + return + if not filename: + expected = [] + elif origname != spec.name and not origname.startswith('<'): + expected = [] + else: + expected = [os.path.dirname(filename)] self.assertListEqual(spec.submodule_search_locations, expected) def test_module(self): diff --git a/Lib/test/test_importlib/frozen/test_loader.py b/Lib/test/test_importlib/frozen/test_loader.py index d6f39fa98a6f1..f1ccb8a188aca 100644 --- a/Lib/test/test_importlib/frozen/test_loader.py +++ b/Lib/test/test_importlib/frozen/test_loader.py @@ -3,10 +3,11 @@ machinery = util.import_importlib('importlib.machinery') -from test.support import captured_stdout, import_helper +from test.support import captured_stdout, import_helper, STDLIB_DIR import _imp import contextlib import marshal +import os.path import types import unittest import warnings @@ -30,20 +31,27 @@ def fresh(name, *, oldapi=False): yield +def resolve_stdlib_file(name, ispkg=False): + assert name + if ispkg: + return os.path.join(STDLIB_DIR, *name.split('.'), '__init__.py') + else: + return os.path.join(STDLIB_DIR, *name.split('.')) + '.py' + + class ExecModuleTests(abc.LoaderTests): def exec_module(self, name, origname=None): with import_helper.frozen_modules(): is_package = self.machinery.FrozenImporter.is_package(name) - code = _imp.get_frozen_object(name) spec = self.machinery.ModuleSpec( name, self.machinery.FrozenImporter, origin='frozen', is_package=is_package, loader_state=types.SimpleNamespace( - data=marshal.dumps(code), origname=origname or name, + filename=resolve_stdlib_file(origname or name, is_package), ), ) module = types.ModuleType(name) @@ -68,7 +76,6 @@ def test_module(self): self.assertEqual(getattr(module, attr), value) self.assertEqual(output, 'Hello world!\n') self.assertTrue(hasattr(module, '__spec__')) - self.assertIsNone(module.__spec__.loader_state.data) self.assertEqual(module.__spec__.loader_state.origname, name) def test_package(self): @@ -82,7 +89,6 @@ def test_package(self): name=name, attr=attr, given=attr_value, expected=value)) self.assertEqual(output, 'Hello world!\n') - self.assertIsNone(module.__spec__.loader_state.data) self.assertEqual(module.__spec__.loader_state.origname, name) def test_lacking_parent(self): @@ -139,36 +145,41 @@ def load_module(self, name): def test_module(self): module, stdout = self.load_module('__hello__') + filename = resolve_stdlib_file('__hello__') check = {'__name__': '__hello__', '__package__': '', '__loader__': self.machinery.FrozenImporter, + '__file__': filename, } for attr, value in check.items(): - self.assertEqual(getattr(module, attr), value) + self.assertEqual(getattr(module, attr, None), value) self.assertEqual(stdout.getvalue(), 'Hello world!\n') - self.assertFalse(hasattr(module, '__file__')) def test_package(self): module, stdout = self.load_module('__phello__') + filename = resolve_stdlib_file('__phello__', ispkg=True) + pkgdir = os.path.dirname(filename) check = {'__name__': '__phello__', '__package__': '__phello__', - '__path__': [], + '__path__': [pkgdir], '__loader__': self.machinery.FrozenImporter, + '__file__': filename, } for attr, value in check.items(): - attr_value = getattr(module, attr) + attr_value = getattr(module, attr, None) self.assertEqual(attr_value, value, "for __phello__.%s, %r != %r" % (attr, attr_value, value)) self.assertEqual(stdout.getvalue(), 'Hello world!\n') - self.assertFalse(hasattr(module, '__file__')) def test_lacking_parent(self): with util.uncache('__phello__'): module, stdout = self.load_module('__phello__.spam') + filename = resolve_stdlib_file('__phello__.spam') check = {'__name__': '__phello__.spam', '__package__': '__phello__', '__loader__': self.machinery.FrozenImporter, + '__file__': filename, } for attr, value in check.items(): attr_value = getattr(module, attr) @@ -176,7 +187,6 @@ def test_lacking_parent(self): "for __phello__.spam.%s, %r != %r" % (attr, attr_value, value)) self.assertEqual(stdout.getvalue(), 'Hello world!\n') - self.assertFalse(hasattr(module, '__file__')) def test_module_reuse(self): with fresh('__hello__', oldapi=True): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-21-02.bpo-21736.RI47BU.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-21-02.bpo-21736.RI47BU.rst new file mode 100644 index 0000000000000..8396a49c3cf73 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-01-09-21-02.bpo-21736.RI47BU.rst @@ -0,0 +1,9 @@ +Frozen stdlib modules now have ``__file__`` to the .py file they would +otherwise be loaded from, if possible. For packages, ``__path__`` now has +the correct entry instead of being an empty list, which allows unfrozen +submodules to be imported. These are set only if the stdlib directory is +known when the runtime is initialized. Note that the file at ``__file__`` +is not guaranteed to exist. None of this affects non-stdlib frozen modules +nor, for now, frozen modules imported using +``PyImport_ImportFrozenModule()``. Also, at the moment ``co_filename`` is +not updated for the module. diff --git a/Python/clinic/import.c.h b/Python/clinic/import.c.h index dfb59de3b5ce8..6052316cdd8e3 100644 --- a/Python/clinic/import.c.h +++ b/Python/clinic/import.c.h @@ -170,7 +170,7 @@ _imp_init_frozen(PyObject *module, PyObject *arg) } PyDoc_STRVAR(_imp_find_frozen__doc__, -"find_frozen($module, name, /)\n" +"find_frozen($module, name, /, *, withdata=False)\n" "--\n" "\n" "Return info about the corresponding frozen module (if there is one) or None.\n" @@ -184,26 +184,43 @@ PyDoc_STRVAR(_imp_find_frozen__doc__, " the module\'s current name)"); #define _IMP_FIND_FROZEN_METHODDEF \ - {"find_frozen", (PyCFunction)_imp_find_frozen, METH_O, _imp_find_frozen__doc__}, + {"find_frozen", (PyCFunction)(void(*)(void))_imp_find_frozen, METH_FASTCALL|METH_KEYWORDS, _imp_find_frozen__doc__}, static PyObject * -_imp_find_frozen_impl(PyObject *module, PyObject *name); +_imp_find_frozen_impl(PyObject *module, PyObject *name, int withdata); static PyObject * -_imp_find_frozen(PyObject *module, PyObject *arg) +_imp_find_frozen(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "withdata", NULL}; + static _PyArg_Parser _parser = {NULL, _keywords, "find_frozen", 0}; + PyObject *argsbuf[2]; + Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 1; PyObject *name; + int withdata = 0; - if (!PyUnicode_Check(arg)) { - _PyArg_BadArgument("find_frozen", "argument", "str", arg); + args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); + if (!args) { goto exit; } - if (PyUnicode_READY(arg) == -1) { + if (!PyUnicode_Check(args[0])) { + _PyArg_BadArgument("find_frozen", "argument 1", "str", args[0]); goto exit; } - name = arg; - return_value = _imp_find_frozen_impl(module, name); + if (PyUnicode_READY(args[0]) == -1) { + goto exit; + } + name = args[0]; + if (!noptargs) { + goto skip_optional_kwonly; + } + withdata = PyObject_IsTrue(args[1]); + if (withdata < 0) { + goto exit; + } +skip_optional_kwonly: + return_value = _imp_find_frozen_impl(module, name, withdata); exit: return return_value; @@ -548,4 +565,4 @@ _imp_source_hash(PyObject *module, PyObject *const *args, Py_ssize_t nargs, PyOb #ifndef _IMP_EXEC_DYNAMIC_METHODDEF #define _IMP_EXEC_DYNAMIC_METHODDEF #endif /* !defined(_IMP_EXEC_DYNAMIC_METHODDEF) */ -/*[clinic end generated code: output=8c8dd08158f9ac7c input=a9049054013a1b77]*/ +/*[clinic end generated code: output=adcf787969a11353 input=a9049054013a1b77]*/ diff --git a/Python/import.c b/Python/import.c index 4bc1e518bf525..f2160928c4fa7 100644 --- a/Python/import.c +++ b/Python/import.c @@ -2049,6 +2049,8 @@ _imp.find_frozen name: unicode / + * + withdata: bool = False Return info about the corresponding frozen module (if there is one) or None. @@ -2062,8 +2064,8 @@ The returned info (a 2-tuple): [clinic start generated code]*/ static PyObject * -_imp_find_frozen_impl(PyObject *module, PyObject *name) -/*[clinic end generated code: output=3fd17da90d417e4e input=6aa7b9078a89280a]*/ +_imp_find_frozen_impl(PyObject *module, PyObject *name, int withdata) +/*[clinic end generated code: output=8c1c3c7f925397a5 input=22a8847c201542fd]*/ { struct frozen_info info; frozen_status status = find_frozen(name, &info); @@ -2078,9 +2080,12 @@ _imp_find_frozen_impl(PyObject *module, PyObject *name) return NULL; } - PyObject *data = PyBytes_FromStringAndSize(info.data, info.size); - if (data == NULL) { - return NULL; + PyObject *data = NULL; + if (withdata) { + data = PyMemoryView_FromMemory((char *)info.data, info.size, PyBUF_READ); + if (data == NULL) { + return NULL; + } } PyObject *origname = NULL; @@ -2092,11 +2097,11 @@ _imp_find_frozen_impl(PyObject *module, PyObject *name) } } - PyObject *result = PyTuple_Pack(3, data, + PyObject *result = PyTuple_Pack(3, data ? data : Py_None, info.is_package ? Py_True : Py_False, origname ? origname : Py_None); Py_XDECREF(origname); - Py_DECREF(data); + Py_XDECREF(data); return result; } @@ -2115,15 +2120,14 @@ _imp_get_frozen_object_impl(PyObject *module, PyObject *name, PyObject *dataobj) /*[clinic end generated code: output=54368a673a35e745 input=034bdb88f6460b7b]*/ { - struct frozen_info info; - if (PyBytes_Check(dataobj)) { - info.nameobj = name; - info.data = PyBytes_AS_STRING(dataobj); - info.size = PyBytes_Size(dataobj); - if (info.size == 0) { - set_frozen_error(FROZEN_INVALID, name); + struct frozen_info info = {0}; + Py_buffer buf = {0}; + if (PyObject_CheckBuffer(dataobj)) { + if (PyObject_GetBuffer(dataobj, &buf, PyBUF_READ) != 0) { return NULL; } + info.data = (const char *)buf.buf; + info.size = buf.len; } else if (dataobj != Py_None) { _PyArg_BadArgument("get_frozen_object", "argument 2", "bytes", dataobj); @@ -2136,7 +2140,20 @@ _imp_get_frozen_object_impl(PyObject *module, PyObject *name, return NULL; } } - return unmarshal_frozen_code(&info); + + if (info.nameobj == NULL) { + info.nameobj = name; + } + if (info.size == 0) { + set_frozen_error(FROZEN_INVALID, name); + return NULL; + } + + PyObject *codeobj = unmarshal_frozen_code(&info); + if (dataobj != Py_None) { + PyBuffer_Release(&buf); + } + return codeobj; } /*[clinic input] From webhook-mailer at python.org Thu Oct 14 17:41:15 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 21:41:15 -0000 Subject: [Python-checkins] bpo-35134: Add Include/cpython/floatobject.h (GH-28957) Message-ID: https://github.com/python/cpython/commit/0a883a76cda8205023c52211968bcf87bd47fd6e commit: 0a883a76cda8205023c52211968bcf87bd47fd6e branch: main author: Victor Stinner committer: vstinner date: 2021-10-14T23:41:06+02:00 summary: bpo-35134: Add Include/cpython/floatobject.h (GH-28957) Split Include/floatobject.h into sub-files: add Include/cpython/floatobject.h and Include/internal/pycore_floatobject.h. files: A Include/cpython/floatobject.h A Include/internal/pycore_floatobject.h M Include/floatobject.h M Makefile.pre.in M Modules/_ctypes/cfield.c M Modules/_pickle.c M Modules/_struct.c M Modules/arraymodule.c M Objects/floatobject.c M Objects/object.c M Objects/stringlib/unicode_format.h M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/marshal.c diff --git a/Include/cpython/floatobject.h b/Include/cpython/floatobject.h new file mode 100644 index 0000000000000..fffd468690274 --- /dev/null +++ b/Include/cpython/floatobject.h @@ -0,0 +1,12 @@ +#ifndef Py_CPYTHON_FLOATOBJECT_H +# error "this header file must not be included directly" +#endif + +typedef struct { + PyObject_HEAD + double ob_fval; +} PyFloatObject; + +// Macro version of PyFloat_AsDouble() trading safety for speed. +// It doesn't check if op is a double object. +#define PyFloat_AS_DOUBLE(op) (((PyFloatObject *)(op))->ob_fval) diff --git a/Include/floatobject.h b/Include/floatobject.h index e994aa8f29da4..3b6ca478eaef2 100644 --- a/Include/floatobject.h +++ b/Include/floatobject.h @@ -11,106 +11,44 @@ PyFloatObject represents a (double precision) floating point number. extern "C" { #endif -#ifndef Py_LIMITED_API -typedef struct { - PyObject_HEAD - double ob_fval; -} PyFloatObject; -#endif - PyAPI_DATA(PyTypeObject) PyFloat_Type; #define PyFloat_Check(op) PyObject_TypeCheck(op, &PyFloat_Type) #define PyFloat_CheckExact(op) Py_IS_TYPE(op, &PyFloat_Type) #ifdef Py_NAN -#define Py_RETURN_NAN return PyFloat_FromDouble(Py_NAN) +# define Py_RETURN_NAN return PyFloat_FromDouble(Py_NAN) #endif -#define Py_RETURN_INF(sign) do \ - if (copysign(1., sign) == 1.) { \ - return PyFloat_FromDouble(Py_HUGE_VAL); \ - } else { \ - return PyFloat_FromDouble(-Py_HUGE_VAL); \ +#define Py_RETURN_INF(sign) \ + do { \ + if (copysign(1., sign) == 1.) { \ + return PyFloat_FromDouble(Py_HUGE_VAL); \ + } \ + else { \ + return PyFloat_FromDouble(-Py_HUGE_VAL); \ + } \ } while(0) PyAPI_FUNC(double) PyFloat_GetMax(void); PyAPI_FUNC(double) PyFloat_GetMin(void); -PyAPI_FUNC(PyObject *) PyFloat_GetInfo(void); +PyAPI_FUNC(PyObject*) PyFloat_GetInfo(void); /* Return Python float from string PyObject. */ -PyAPI_FUNC(PyObject *) PyFloat_FromString(PyObject*); +PyAPI_FUNC(PyObject*) PyFloat_FromString(PyObject*); /* Return Python float from C double. */ -PyAPI_FUNC(PyObject *) PyFloat_FromDouble(double); +PyAPI_FUNC(PyObject*) PyFloat_FromDouble(double); /* Extract C double from Python float. The macro version trades safety for speed. */ -PyAPI_FUNC(double) PyFloat_AsDouble(PyObject *); -#ifndef Py_LIMITED_API -#define PyFloat_AS_DOUBLE(op) (((PyFloatObject *)(op))->ob_fval) -#endif +PyAPI_FUNC(double) PyFloat_AsDouble(PyObject*); #ifndef Py_LIMITED_API -/* _PyFloat_{Pack,Unpack}{4,8} - * - * The struct and pickle (at least) modules need an efficient platform- - * independent way to store floating-point values as byte strings. - * The Pack routines produce a string from a C double, and the Unpack - * routines produce a C double from such a string. The suffix (4 or 8) - * specifies the number of bytes in the string. - * - * On platforms that appear to use (see _PyFloat_Init()) IEEE-754 formats - * these functions work by copying bits. On other platforms, the formats the - * 4- byte format is identical to the IEEE-754 single precision format, and - * the 8-byte format to the IEEE-754 double precision format, although the - * packing of INFs and NaNs (if such things exist on the platform) isn't - * handled correctly, and attempting to unpack a string containing an IEEE - * INF or NaN will raise an exception. - * - * On non-IEEE platforms with more precision, or larger dynamic range, than - * 754 supports, not all values can be packed; on non-IEEE platforms with less - * precision, or smaller dynamic range, not all values can be unpacked. What - * happens in such cases is partly accidental (alas). - */ - -/* The pack routines write 2, 4 or 8 bytes, starting at p. le is a bool - * argument, true if you want the string in little-endian format (exponent - * last, at p+1, p+3 or p+7), false if you want big-endian format (exponent - * first, at p). - * Return value: 0 if all is OK, -1 if error (and an exception is - * set, most likely OverflowError). - * There are two problems on non-IEEE platforms: - * 1): What this does is undefined if x is a NaN or infinity. - * 2): -0.0 and +0.0 produce the same string. - */ -PyAPI_FUNC(int) _PyFloat_Pack2(double x, unsigned char *p, int le); -PyAPI_FUNC(int) _PyFloat_Pack4(double x, unsigned char *p, int le); -PyAPI_FUNC(int) _PyFloat_Pack8(double x, unsigned char *p, int le); - -/* The unpack routines read 2, 4 or 8 bytes, starting at p. le is a bool - * argument, true if the string is in little-endian format (exponent - * last, at p+1, p+3 or p+7), false if big-endian (exponent first, at p). - * Return value: The unpacked double. On error, this is -1.0 and - * PyErr_Occurred() is true (and an exception is set, most likely - * OverflowError). Note that on a non-IEEE platform this will refuse - * to unpack a string that represents a NaN or infinity. - */ -PyAPI_FUNC(double) _PyFloat_Unpack2(const unsigned char *p, int le); -PyAPI_FUNC(double) _PyFloat_Unpack4(const unsigned char *p, int le); -PyAPI_FUNC(double) _PyFloat_Unpack8(const unsigned char *p, int le); - -PyAPI_FUNC(void) _PyFloat_DebugMallocStats(FILE* out); - -/* Format the object based on the format_spec, as defined in PEP 3101 - (Advanced String Formatting). */ -PyAPI_FUNC(int) _PyFloat_FormatAdvancedWriter( - _PyUnicodeWriter *writer, - PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); -#endif /* Py_LIMITED_API */ +# define Py_CPYTHON_FLOATOBJECT_H +# include "cpython/floatobject.h" +# undef Py_CPYTHON_FLOATOBJECT_H +#endif #ifdef __cplusplus } diff --git a/Include/internal/pycore_floatobject.h b/Include/internal/pycore_floatobject.h new file mode 100644 index 0000000000000..18227c9e36925 --- /dev/null +++ b/Include/internal/pycore_floatobject.h @@ -0,0 +1,75 @@ +#ifndef Py_INTERNAL_FLOATOBJECT_H +#define Py_INTERNAL_FLOATOBJECT_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +/* _PyFloat_{Pack,Unpack}{4,8} + * + * The struct and pickle (at least) modules need an efficient platform- + * independent way to store floating-point values as byte strings. + * The Pack routines produce a string from a C double, and the Unpack + * routines produce a C double from such a string. The suffix (4 or 8) + * specifies the number of bytes in the string. + * + * On platforms that appear to use (see _PyFloat_Init()) IEEE-754 formats + * these functions work by copying bits. On other platforms, the formats the + * 4- byte format is identical to the IEEE-754 single precision format, and + * the 8-byte format to the IEEE-754 double precision format, although the + * packing of INFs and NaNs (if such things exist on the platform) isn't + * handled correctly, and attempting to unpack a string containing an IEEE + * INF or NaN will raise an exception. + * + * On non-IEEE platforms with more precision, or larger dynamic range, than + * 754 supports, not all values can be packed; on non-IEEE platforms with less + * precision, or smaller dynamic range, not all values can be unpacked. What + * happens in such cases is partly accidental (alas). + */ + +/* The pack routines write 2, 4 or 8 bytes, starting at p. le is a bool + * argument, true if you want the string in little-endian format (exponent + * last, at p+1, p+3 or p+7), false if you want big-endian format (exponent + * first, at p). + * Return value: 0 if all is OK, -1 if error (and an exception is + * set, most likely OverflowError). + * There are two problems on non-IEEE platforms: + * 1): What this does is undefined if x is a NaN or infinity. + * 2): -0.0 and +0.0 produce the same string. + */ +PyAPI_FUNC(int) _PyFloat_Pack2(double x, unsigned char *p, int le); +PyAPI_FUNC(int) _PyFloat_Pack4(double x, unsigned char *p, int le); +PyAPI_FUNC(int) _PyFloat_Pack8(double x, unsigned char *p, int le); + +/* The unpack routines read 2, 4 or 8 bytes, starting at p. le is a bool + * argument, true if the string is in little-endian format (exponent + * last, at p+1, p+3 or p+7), false if big-endian (exponent first, at p). + * Return value: The unpacked double. On error, this is -1.0 and + * PyErr_Occurred() is true (and an exception is set, most likely + * OverflowError). Note that on a non-IEEE platform this will refuse + * to unpack a string that represents a NaN or infinity. + */ +PyAPI_FUNC(double) _PyFloat_Unpack2(const unsigned char *p, int le); +PyAPI_FUNC(double) _PyFloat_Unpack4(const unsigned char *p, int le); +PyAPI_FUNC(double) _PyFloat_Unpack8(const unsigned char *p, int le); + + +PyAPI_FUNC(void) _PyFloat_DebugMallocStats(FILE* out); + + +/* Format the object based on the format_spec, as defined in PEP 3101 + (Advanced String Formatting). */ +PyAPI_FUNC(int) _PyFloat_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); + +#ifdef __cplusplus +} +#endif +#endif /* !Py_INTERNAL_FLOATOBJECT_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index 041bc63e2fb8e..30b025e7efb83 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1208,6 +1208,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/dictobject.h \ $(srcdir)/Include/cpython/fileobject.h \ $(srcdir)/Include/cpython/fileutils.h \ + $(srcdir)/Include/cpython/floatobject.h \ $(srcdir)/Include/cpython/frameobject.h \ $(srcdir)/Include/cpython/import.h \ $(srcdir)/Include/cpython/initconfig.h \ @@ -1250,6 +1251,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_dict.h \ $(srcdir)/Include/internal/pycore_dtoa.h \ $(srcdir)/Include/internal/pycore_fileutils.h \ + $(srcdir)/Include/internal/pycore_floatobject.h \ $(srcdir)/Include/internal/pycore_format.h \ $(srcdir)/Include/internal/pycore_getopt.h \ $(srcdir)/Include/internal/pycore_gil.h \ diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index b7585bc8a9fff..2cfd657028aca 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -6,6 +6,7 @@ #include "pycore_bitutils.h" // _Py_bswap32() #include "pycore_call.h" // _PyObject_CallNoArgs() +#include "pycore_floatobject.h" // _PyFloat_Pack8() #include #include "ctypes.h" diff --git a/Modules/_pickle.c b/Modules/_pickle.c index b5131696981dc..53ab57111967c 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -9,6 +9,7 @@ #endif #include "Python.h" +#include "pycore_floatobject.h" // _PyFloat_Pack8() #include "pycore_moduleobject.h" // _PyModule_GetState() #include "structmember.h" // PyMemberDef diff --git a/Modules/_struct.c b/Modules/_struct.c index 69de080f4388c..a8003a90b2682 100644 --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -6,6 +6,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_floatobject.h" // _PyFloat_Unpack2() #include "pycore_moduleobject.h" // _PyModule_GetState() #include "structmember.h" // PyMemberDef #include diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c index c9b6a92c22749..5e57fe116059d 100644 --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -5,6 +5,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_floatobject.h" // _PyFloat_Unpack4() #include "pycore_moduleobject.h" // _PyModule_GetState() #include "structmember.h" // PyMemberDef #include // offsetof() diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 5a8113eca8bd0..1be31e38d4935 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -5,6 +5,7 @@ #include "Python.h" #include "pycore_dtoa.h" // _Py_dg_dtoa() +#include "pycore_floatobject.h" // _PyFloat_FormatAdvancedWriter() #include "pycore_interp.h" // _PyInterpreterState.float_state #include "pycore_long.h" // _PyLong_GetOne() #include "pycore_object.h" // _PyObject_Init() diff --git a/Objects/object.c b/Objects/object.c index 14c85c233df11..589dd6647318d 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -6,6 +6,7 @@ #include "pycore_ceval.h" // _Py_EnterRecursiveCall() #include "pycore_context.h" #include "pycore_dict.h" +#include "pycore_floatobject.h" // _PyFloat_DebugMallocStats() #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() #include "pycore_object.h" // _PyType_CheckConsistency() #include "pycore_pyerrors.h" // _PyErr_Occurred() diff --git a/Objects/stringlib/unicode_format.h b/Objects/stringlib/unicode_format.h index 7152ec6ebe712..a4eea7b91988b 100644 --- a/Objects/stringlib/unicode_format.h +++ b/Objects/stringlib/unicode_format.h @@ -2,6 +2,8 @@ unicode_format.h -- implementation of str.format(). */ +#include "pycore_floatobject.h" // _PyFloat_FormatAdvancedWriter() + /************************************************************************/ /*********** Global data structures and forward declarations *********/ /************************************************************************/ diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index d0e1b52a4927c..f688e8a323245 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -132,6 +132,7 @@ + @@ -188,6 +189,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 9370f405132c3..eb72c38a7b203 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -393,6 +393,9 @@ Include\cpython + + Include\cpython + Include\cpython @@ -522,6 +525,9 @@ Include\internal + + Include\internal + Include\internal diff --git a/Python/marshal.c b/Python/marshal.c index c8a48a55dee0f..e9ad566b71b10 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -11,6 +11,7 @@ #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_code.h" // _PyCode_New() +#include "pycore_floatobject.h" // _PyFloat_Pack8() #include "pycore_hashtable.h" // _Py_hashtable_t #include "longintrepr.h" #include "code.h" From webhook-mailer at python.org Thu Oct 14 18:20:40 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 22:20:40 -0000 Subject: [Python-checkins] bpo-45474: Fix the limited C API of marshal.h (GH-28956) Message-ID: https://github.com/python/cpython/commit/af1083e975b9627a5c97013d3a2b9aef0e4b333c commit: af1083e975b9627a5c97013d3a2b9aef0e4b333c branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T00:20:33+02:00 summary: bpo-45474: Fix the limited C API of marshal.h (GH-28956) Remove two functions from the limited C API: * PyMarshal_WriteLongToFile() * PyMarshal_WriteObjectToFile() The PEP 384 excludes functions expecting "FILE*" from the stable ABI. Remove also the Py_MARSHAL_VERSION macro from the limited C API. files: A Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst M Doc/whatsnew/3.11.rst M Include/marshal.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 48d454d9aac99..e8d64a80a69a3 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -613,3 +613,13 @@ Removed * Remove the ``Py_FORCE_DOUBLE()`` macro. It was used by the ``Py_IS_INFINITY()`` macro. (Contributed by Victor Stinner in :issue:`45440`.) + +* Remove two functions from the limited C API: + + * :c:func:`PyMarshal_WriteLongToFile` + * :c:func:`PyMarshal_WriteObjectToFile` + + The :pep:`384` excludes functions expecting ``FILE*`` from the stable ABI. + + Remove also the ``Py_MARSHAL_VERSION`` macro from the limited C API. + (Contributed by Victor Stinner in :issue:`45474`.) diff --git a/Include/marshal.h b/Include/marshal.h index 09d9337e57b0a..36ef6a779ec2c 100644 --- a/Include/marshal.h +++ b/Include/marshal.h @@ -7,20 +7,21 @@ extern "C" { #endif -#define Py_MARSHAL_VERSION 4 - -PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); -PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); +PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromString(const char *, + Py_ssize_t); PyAPI_FUNC(PyObject *) PyMarshal_WriteObjectToString(PyObject *, int); #ifndef Py_LIMITED_API +#define Py_MARSHAL_VERSION 4 + PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *); PyAPI_FUNC(int) PyMarshal_ReadShortFromFile(FILE *); PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromFile(FILE *); PyAPI_FUNC(PyObject *) PyMarshal_ReadLastObjectFromFile(FILE *); + +PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); +PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); #endif -PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromString(const char *, - Py_ssize_t); #ifdef __cplusplus } diff --git a/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst b/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst new file mode 100644 index 0000000000000..90bf498579c11 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst @@ -0,0 +1,10 @@ +Remove two functions from the limited C API: + +* :c:func:`PyMarshal_WriteLongToFile` +* :c:func:`PyMarshal_WriteObjectToFile` + +The :pep:`384` excludes functions expecting ``FILE*`` from the stable ABI. + +Remove also the ``Py_MARSHAL_VERSION`` macro from the limited C API. + +Patch by Victor Stinner. From webhook-mailer at python.org Thu Oct 14 19:09:14 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 23:09:14 -0000 Subject: [Python-checkins] bpo-45434: Limited Python.h no longer includes stdio.h (GH-28960) Message-ID: https://github.com/python/cpython/commit/284994762d820d8e09cc019f8f7c4bc501e37dd4 commit: 284994762d820d8e09cc019f8f7c4bc501e37dd4 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T01:09:06+02:00 summary: bpo-45434: Limited Python.h no longer includes stdio.h (GH-28960) The header file no longer includes if the Py_LIMITED_API macro is defined. files: A Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst M Doc/whatsnew/3.11.rst M Include/Python.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index e8d64a80a69a3..a6e30c77c55a0 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -565,6 +565,13 @@ Porting to Python 3.11 ``exit()`` and ``abort()``. (Contributed by Victor Stinner in :issue:`45434`.) +* The ```` header file no longer includes ```` if the + ``Py_LIMITED_API`` macro is defined. Functions expecting ``FILE*`` are + excluded from the limited C API (:pep:`384`). C extensions using + ```` must now include it explicitly. The system ```` + header provides functions like ``printf()`` and ``fopen()``. + (Contributed by Victor Stinner in :issue:`45434`.) + Deprecated ---------- diff --git a/Include/Python.h b/Include/Python.h index 4f62103e30276..dc5c9b8e6384e 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -16,12 +16,10 @@ # define _SGI_MP_SOURCE #endif -#include // NULL, FILE* -#ifndef NULL -# error "Python.h requires that stdio.h define NULL." -#endif - #include // memcpy() +#ifndef Py_LIMITED_API +# include // FILE* +#endif #ifdef HAVE_ERRNO_H # include // errno #endif @@ -29,8 +27,7 @@ # include #endif #ifdef HAVE_STDDEF_H - // For size_t -# include +# include // size_t #endif #include // assert() diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst new file mode 100644 index 0000000000000..4a06635d179d9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst @@ -0,0 +1,5 @@ +The ```` header file no longer includes ```` if the +``Py_LIMITED_API`` macro is defined. Functions expecting ``FILE*`` are excluded +from the limited C API (:pep:`384`). C extensions using ```` must now +include it explicitly. +Patch by Victor Stinner. From webhook-mailer at python.org Thu Oct 14 19:49:41 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 23:49:41 -0000 Subject: [Python-checkins] bpo-41710: Fix What's New Entry credit (GH-28962) Message-ID: https://github.com/python/cpython/commit/03bbc6066ff40c62edd57612be9150dcf1b123c8 commit: 03bbc6066ff40c62edd57612be9150dcf1b123c8 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T01:49:32+02:00 summary: bpo-41710: Fix What's New Entry credit (GH-28962) Fix bad copy/paste. files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a6e30c77c55a0..0647774f400d6 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -252,7 +252,7 @@ threading the monotonic clock (:data:`time.CLOCK_MONOTONIC`) for the timeout, rather than using the system clock (:data:`time.CLOCK_REALTIME`), to not be affected by system clock changes. - (Contributed by Livius and Victor Stinner in :issue:`41710`.) + (Contributed by Victor Stinner in :issue:`41710`.) time ---- From webhook-mailer at python.org Thu Oct 14 19:50:13 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 23:50:13 -0000 Subject: [Python-checkins] bpo-45434: Remove useless space in includes (GH-28963) Message-ID: https://github.com/python/cpython/commit/7076bef8ba8836a19d5033f4ceb8eb9837de2301 commit: 7076bef8ba8836a19d5033f4ceb8eb9837de2301 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T01:50:04+02:00 summary: bpo-45434: Remove useless space in includes (GH-28963) Micro-optimize spaces! files: M Include/abstract.h M Include/bytearrayobject.h M Include/bytesobject.h M Include/ceval.h M Include/code.h M Include/dictobject.h M Include/fileobject.h M Include/fileutils.h M Include/frameobject.h M Include/import.h M Include/interpreteridobject.h M Include/listobject.h M Include/methodobject.h M Include/object.h M Include/objimpl.h M Include/pyerrors.h M Include/pylifecycle.h M Include/pymem.h M Include/pystate.h M Include/pythonrun.h M Include/sysmodule.h M Include/traceback.h M Include/tupleobject.h M Include/unicodeobject.h diff --git a/Include/abstract.h b/Include/abstract.h index 9eaab6b2e054c..9e06fbbb74913 100644 --- a/Include/abstract.h +++ b/Include/abstract.h @@ -863,7 +863,7 @@ PyAPI_FUNC(int) PyObject_IsSubclass(PyObject *object, PyObject *typeorclass); #ifndef Py_LIMITED_API # define Py_CPYTHON_ABSTRACTOBJECT_H -# include "cpython/abstract.h" +# include "cpython/abstract.h" # undef Py_CPYTHON_ABSTRACTOBJECT_H #endif diff --git a/Include/bytearrayobject.h b/Include/bytearrayobject.h index 1a834474dde7c..ae2bde1c30356 100644 --- a/Include/bytearrayobject.h +++ b/Include/bytearrayobject.h @@ -34,7 +34,7 @@ PyAPI_FUNC(int) PyByteArray_Resize(PyObject *, Py_ssize_t); #ifndef Py_LIMITED_API # define Py_CPYTHON_BYTEARRAYOBJECT_H -# include "cpython/bytearrayobject.h" +# include "cpython/bytearrayobject.h" # undef Py_CPYTHON_BYTEARRAYOBJECT_H #endif diff --git a/Include/bytesobject.h b/Include/bytesobject.h index bcb1a5942c68f..4c4dc40d705d7 100644 --- a/Include/bytesobject.h +++ b/Include/bytesobject.h @@ -59,7 +59,7 @@ PyAPI_FUNC(int) PyBytes_AsStringAndSize( #ifndef Py_LIMITED_API # define Py_CPYTHON_BYTESOBJECT_H -# include "cpython/bytesobject.h" +# include "cpython/bytesobject.h" # undef Py_CPYTHON_BYTESOBJECT_H #endif diff --git a/Include/ceval.h b/Include/ceval.h index 0f687666e2bcc..cf8c5b17be589 100644 --- a/Include/ceval.h +++ b/Include/ceval.h @@ -148,7 +148,7 @@ PyAPI_FUNC(void) PyEval_ReleaseThread(PyThreadState *tstate); #ifndef Py_LIMITED_API # define Py_CPYTHON_CEVAL_H -# include "cpython/ceval.h" +# include "cpython/ceval.h" # undef Py_CPYTHON_CEVAL_H #endif diff --git a/Include/code.h b/Include/code.h index b9e23eb816529..2dea3c2610531 100644 --- a/Include/code.h +++ b/Include/code.h @@ -10,7 +10,7 @@ typedef struct PyCodeObject PyCodeObject; #ifndef Py_LIMITED_API # define Py_CPYTHON_CODE_H -# include "cpython/code.h" +# include "cpython/code.h" # undef Py_CPYTHON_CODE_H #endif diff --git a/Include/dictobject.h b/Include/dictobject.h index da5a36ba07f32..a6233d8ae2512 100644 --- a/Include/dictobject.h +++ b/Include/dictobject.h @@ -87,7 +87,7 @@ PyAPI_DATA(PyTypeObject) PyDictRevIterValue_Type; #ifndef Py_LIMITED_API # define Py_CPYTHON_DICTOBJECT_H -# include "cpython/dictobject.h" +# include "cpython/dictobject.h" # undef Py_CPYTHON_DICTOBJECT_H #endif diff --git a/Include/fileobject.h b/Include/fileobject.h index 6ec2994aa859b..4c983e7b5daa8 100644 --- a/Include/fileobject.h +++ b/Include/fileobject.h @@ -39,7 +39,7 @@ PyAPI_DATA(int) Py_UTF8Mode; #ifndef Py_LIMITED_API # define Py_CPYTHON_FILEOBJECT_H -# include "cpython/fileobject.h" +# include "cpython/fileobject.h" # undef Py_CPYTHON_FILEOBJECT_H #endif diff --git a/Include/fileutils.h b/Include/fileutils.h index 16f3b635deed8..ba5acc84fcb18 100644 --- a/Include/fileutils.h +++ b/Include/fileutils.h @@ -16,7 +16,7 @@ PyAPI_FUNC(char*) Py_EncodeLocale( #ifndef Py_LIMITED_API # define Py_CPYTHON_FILEUTILS_H -# include "cpython/fileutils.h" +# include "cpython/fileutils.h" # undef Py_CPYTHON_FILEUTILS_H #endif diff --git a/Include/frameobject.h b/Include/frameobject.h index c118af1201a4c..adb628f6314fc 100644 --- a/Include/frameobject.h +++ b/Include/frameobject.h @@ -10,7 +10,7 @@ extern "C" { #ifndef Py_LIMITED_API # define Py_CPYTHON_FRAMEOBJECT_H -# include "cpython/frameobject.h" +# include "cpython/frameobject.h" # undef Py_CPYTHON_FRAMEOBJECT_H #endif diff --git a/Include/import.h b/Include/import.h index aeef3efd0bcee..a87677bb10c7f 100644 --- a/Include/import.h +++ b/Include/import.h @@ -88,7 +88,7 @@ PyAPI_FUNC(int) PyImport_AppendInittab( #ifndef Py_LIMITED_API # define Py_CPYTHON_IMPORT_H -# include "cpython/import.h" +# include "cpython/import.h" # undef Py_CPYTHON_IMPORT_H #endif diff --git a/Include/interpreteridobject.h b/Include/interpreteridobject.h index e744fcdc9ff18..8432632f339e9 100644 --- a/Include/interpreteridobject.h +++ b/Include/interpreteridobject.h @@ -7,7 +7,7 @@ extern "C" { #ifndef Py_LIMITED_API # define Py_CPYTHON_INTERPRETERIDOBJECT_H -# include "cpython/interpreteridobject.h" +# include "cpython/interpreteridobject.h" # undef Py_CPYTHON_INTERPRETERIDOBJECT_H #endif diff --git a/Include/listobject.h b/Include/listobject.h index 2a8a25525d1d7..eff42c188f1ff 100644 --- a/Include/listobject.h +++ b/Include/listobject.h @@ -42,7 +42,7 @@ PyAPI_FUNC(PyObject *) PyList_AsTuple(PyObject *); #ifndef Py_LIMITED_API # define Py_CPYTHON_LISTOBJECT_H -# include "cpython/listobject.h" +# include "cpython/listobject.h" # undef Py_CPYTHON_LISTOBJECT_H #endif diff --git a/Include/methodobject.h b/Include/methodobject.h index 9ffe8e1a3ddfc..1be5873a30569 100644 --- a/Include/methodobject.h +++ b/Include/methodobject.h @@ -103,11 +103,9 @@ PyAPI_FUNC(PyObject *) PyCMethod_New(PyMethodDef *, PyObject *, #ifndef Py_LIMITED_API - -#define Py_CPYTHON_METHODOBJECT_H -#include "cpython/methodobject.h" -#undef Py_CPYTHON_METHODOBJECT_H - +# define Py_CPYTHON_METHODOBJECT_H +# include "cpython/methodobject.h" +# undef Py_CPYTHON_METHODOBJECT_H #endif #ifdef __cplusplus diff --git a/Include/object.h b/Include/object.h index 7f050b80aaff1..33df303a44eb7 100644 --- a/Include/object.h +++ b/Include/object.h @@ -724,7 +724,7 @@ times. #ifndef Py_LIMITED_API # define Py_CPYTHON_OBJECT_H -# include "cpython/object.h" +# include "cpython/object.h" # undef Py_CPYTHON_OBJECT_H #endif diff --git a/Include/objimpl.h b/Include/objimpl.h index 450befad679e7..9b98c112ac2cc 100644 --- a/Include/objimpl.h +++ b/Include/objimpl.h @@ -205,7 +205,7 @@ PyAPI_FUNC(int) PyObject_GC_IsFinalized(PyObject *); #ifndef Py_LIMITED_API # define Py_CPYTHON_OBJIMPL_H -# include "cpython/objimpl.h" +# include "cpython/objimpl.h" # undef Py_CPYTHON_OBJIMPL_H #endif diff --git a/Include/pyerrors.h b/Include/pyerrors.h index f5d1c71157718..c6c443a2d7d0f 100644 --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -314,7 +314,7 @@ PyAPI_FUNC(int) PyOS_vsnprintf(char *str, size_t size, const char *format, va_l #ifndef Py_LIMITED_API # define Py_CPYTHON_ERRORS_H -# include "cpython/pyerrors.h" +# include "cpython/pyerrors.h" # undef Py_CPYTHON_ERRORS_H #endif diff --git a/Include/pylifecycle.h b/Include/pylifecycle.h index 9b2dd0868eb25..4aecda235abf7 100644 --- a/Include/pylifecycle.h +++ b/Include/pylifecycle.h @@ -64,7 +64,7 @@ PyAPI_FUNC(PyOS_sighandler_t) PyOS_setsig(int, PyOS_sighandler_t); #ifndef Py_LIMITED_API # define Py_CPYTHON_PYLIFECYCLE_H -# include "cpython/pylifecycle.h" +# include "cpython/pylifecycle.h" # undef Py_CPYTHON_PYLIFECYCLE_H #endif diff --git a/Include/pymem.h b/Include/pymem.h index 66cdb0d2973cd..c15ad10dfcf83 100644 --- a/Include/pymem.h +++ b/Include/pymem.h @@ -93,7 +93,7 @@ PyAPI_FUNC(void) PyMem_Free(void *ptr); #ifndef Py_LIMITED_API # define Py_CPYTHON_PYMEM_H -# include "cpython/pymem.h" +# include "cpython/pymem.h" # undef Py_CPYTHON_PYMEM_H #endif diff --git a/Include/pystate.h b/Include/pystate.h index d81f42a3efe32..b6ee0ede81dea 100644 --- a/Include/pystate.h +++ b/Include/pystate.h @@ -132,7 +132,7 @@ PyAPI_FUNC(PyThreadState *) PyGILState_GetThisThreadState(void); #ifndef Py_LIMITED_API # define Py_CPYTHON_PYSTATE_H -# include "cpython/pystate.h" +# include "cpython/pystate.h" # undef Py_CPYTHON_PYSTATE_H #endif diff --git a/Include/pythonrun.h b/Include/pythonrun.h index b0a2fc3002d37..02715775581c6 100644 --- a/Include/pythonrun.h +++ b/Include/pythonrun.h @@ -34,7 +34,7 @@ PyAPI_FUNC(int) PyOS_CheckStack(void); #ifndef Py_LIMITED_API # define Py_CPYTHON_PYTHONRUN_H -# include "cpython/pythonrun.h" +# include "cpython/pythonrun.h" # undef Py_CPYTHON_PYTHONRUN_H #endif diff --git a/Include/sysmodule.h b/Include/sysmodule.h index 8c8f7c425942a..3463c62230900 100644 --- a/Include/sysmodule.h +++ b/Include/sysmodule.h @@ -31,7 +31,7 @@ PyAPI_FUNC(PyObject *) PySys_GetXOptions(void); #ifndef Py_LIMITED_API # define Py_CPYTHON_SYSMODULE_H -# include "cpython/sysmodule.h" +# include "cpython/sysmodule.h" # undef Py_CPYTHON_SYSMODULE_H #endif diff --git a/Include/traceback.h b/Include/traceback.h index 781e5a6eec4ed..2dfa2ada4f2c3 100644 --- a/Include/traceback.h +++ b/Include/traceback.h @@ -16,7 +16,7 @@ PyAPI_DATA(PyTypeObject) PyTraceBack_Type; #ifndef Py_LIMITED_API # define Py_CPYTHON_TRACEBACK_H -# include "cpython/traceback.h" +# include "cpython/traceback.h" # undef Py_CPYTHON_TRACEBACK_H #endif diff --git a/Include/tupleobject.h b/Include/tupleobject.h index e796a320192c2..dc68e3fc5c6d8 100644 --- a/Include/tupleobject.h +++ b/Include/tupleobject.h @@ -36,7 +36,7 @@ PyAPI_FUNC(PyObject *) PyTuple_Pack(Py_ssize_t, ...); #ifndef Py_LIMITED_API # define Py_CPYTHON_TUPLEOBJECT_H -# include "cpython/tupleobject.h" +# include "cpython/tupleobject.h" # undef Py_CPYTHON_TUPLEOBJECT_H #endif diff --git a/Include/unicodeobject.h b/Include/unicodeobject.h index c65b9228298d7..abce967caff78 100644 --- a/Include/unicodeobject.h +++ b/Include/unicodeobject.h @@ -1043,7 +1043,7 @@ PyAPI_FUNC(int) PyUnicode_IsIdentifier(PyObject *s); #ifndef Py_LIMITED_API # define Py_CPYTHON_UNICODEOBJECT_H -# include "cpython/unicodeobject.h" +# include "cpython/unicodeobject.h" # undef Py_CPYTHON_UNICODEOBJECT_H #endif From webhook-mailer at python.org Thu Oct 14 19:50:37 2021 From: webhook-mailer at python.org (vstinner) Date: Thu, 14 Oct 2021 23:50:37 -0000 Subject: [Python-checkins] po-35134: Move Include/funcobject.h to Include/cpython/ (GH-28958) Message-ID: https://github.com/python/cpython/commit/37b1d607bf0f1a9c1e89b1715349efc24dc180e0 commit: 37b1d607bf0f1a9c1e89b1715349efc24dc180e0 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T01:50:28+02:00 summary: po-35134: Move Include/funcobject.h to Include/cpython/ (GH-28958) Remove redundant "#ifndef Py_LIMITED_API" in funcobject.h. files: A Include/cpython/funcobject.h A Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst D Include/funcobject.h M Doc/whatsnew/3.11.rst M Include/Python.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Tools/scripts/stable_abi.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 0647774f400d6..a45568392fbf9 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -572,6 +572,12 @@ Porting to Python 3.11 header provides functions like ``printf()`` and ``fopen()``. (Contributed by Victor Stinner in :issue:`45434`.) +* The non-limited API file ``funcobject.h`` has been moved to the + ``Include/cpython`` directory. This file must not be included directly, as it + is already included in ``Python.h``: :ref:`Include Files `. If + it has been included directly, consider including ``Python.h`` instead. + (Contributed by Victor Stinner in :issue:`35134`.) + Deprecated ---------- diff --git a/Include/Python.h b/Include/Python.h index dc5c9b8e6384e..e8e061bdf62e8 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -60,7 +60,7 @@ #include "setobject.h" #include "methodobject.h" #include "moduleobject.h" -#include "funcobject.h" +#include "cpython/funcobject.h" #include "classobject.h" #include "fileobject.h" #include "pycapsule.h" diff --git a/Include/funcobject.h b/Include/cpython/funcobject.h similarity index 99% rename from Include/funcobject.h rename to Include/cpython/funcobject.h index 6bc03f57d4cb3..60b702218a1f4 100644 --- a/Include/funcobject.h +++ b/Include/cpython/funcobject.h @@ -1,5 +1,5 @@ - /* Function object interface */ + #ifndef Py_LIMITED_API #ifndef Py_FUNCOBJECT_H #define Py_FUNCOBJECT_H @@ -76,7 +76,6 @@ PyAPI_FUNC(int) PyFunction_SetClosure(PyObject *, PyObject *); PyAPI_FUNC(PyObject *) PyFunction_GetAnnotations(PyObject *); PyAPI_FUNC(int) PyFunction_SetAnnotations(PyObject *, PyObject *); -#ifndef Py_LIMITED_API PyAPI_FUNC(PyObject *) _PyFunction_Vectorcall( PyObject *func, PyObject *const *stack, @@ -84,7 +83,6 @@ PyAPI_FUNC(PyObject *) _PyFunction_Vectorcall( PyObject *kwnames); uint32_t _PyFunction_GetVersionForCurrentState(PyFunctionObject *func); -#endif /* Macros for direct access to these values. Type checks are *not* done, so use with care. */ diff --git a/Makefile.pre.in b/Makefile.pre.in index 30b025e7efb83..a5585c8de8ad0 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1145,7 +1145,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/fileutils.h \ $(srcdir)/Include/floatobject.h \ $(srcdir)/Include/frameobject.h \ - $(srcdir)/Include/funcobject.h \ $(srcdir)/Include/genobject.h \ $(srcdir)/Include/import.h \ $(srcdir)/Include/interpreteridobject.h \ @@ -1210,6 +1209,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/fileutils.h \ $(srcdir)/Include/cpython/floatobject.h \ $(srcdir)/Include/cpython/frameobject.h \ + $(srcdir)/Include/cpython/funcobject.h \ $(srcdir)/Include/cpython/import.h \ $(srcdir)/Include/cpython/initconfig.h \ $(srcdir)/Include/cpython/interpreteridobject.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst new file mode 100644 index 0000000000000..fc12e02b0c580 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst @@ -0,0 +1,3 @@ +Move Include/funcobject.h header file to Include/cpython/funcobject.h. +C extensions should only include the main ```` header. +Patch by Victor Stinner. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index f688e8a323245..dc216e34855df 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -134,6 +134,7 @@ + @@ -169,7 +170,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index eb72c38a7b203..8eeb38871f327 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -108,9 +108,6 @@ Include - - Include - Include @@ -456,6 +453,9 @@ Include\cpython + + Include\cpython + Include\cpython diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 6d7034090f881..6cb310e5a31d5 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -34,7 +34,6 @@ "datetime.h", "dtoa.h", "frameobject.h", - "funcobject.h", "genobject.h", "longintrepr.h", "parsetok.h", From webhook-mailer at python.org Thu Oct 14 20:40:09 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 00:40:09 -0000 Subject: [Python-checkins] bpo-35134: Move Include/cellobject.h to Include/cpython/ (GH-28964) Message-ID: https://github.com/python/cpython/commit/77b24ba505744532d7cfd721b1c92d205e145180 commit: 77b24ba505744532d7cfd721b1c92d205e145180 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T02:39:58+02:00 summary: bpo-35134: Move Include/cellobject.h to Include/cpython/ (GH-28964) files: A Include/cpython/cellobject.h D Include/cellobject.h M Doc/whatsnew/3.11.rst M Include/Python.h M Makefile.pre.in M Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a45568392fbf9..734cf1572fcba 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -572,10 +572,11 @@ Porting to Python 3.11 header provides functions like ``printf()`` and ``fopen()``. (Contributed by Victor Stinner in :issue:`45434`.) -* The non-limited API file ``funcobject.h`` has been moved to the - ``Include/cpython`` directory. This file must not be included directly, as it - is already included in ``Python.h``: :ref:`Include Files `. If - it has been included directly, consider including ``Python.h`` instead. +* The non-limited API files ``cellobject.h`` and ``funcobject.h`` have been + moved to the ``Include/cpython`` directory. These files must not be included + directly, as they are already included in ``Python.h``: :ref:`Include Files + `. If they have been included directly, consider including + ``Python.h`` instead. (Contributed by Victor Stinner in :issue:`35134`.) Deprecated diff --git a/Include/Python.h b/Include/Python.h index e8e061bdf62e8..89f60fe5c9f94 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -68,7 +68,7 @@ #include "pyframe.h" #include "traceback.h" #include "sliceobject.h" -#include "cellobject.h" +#include "cpython/cellobject.h" #include "iterobject.h" #include "genobject.h" #include "descrobject.h" diff --git a/Include/cellobject.h b/Include/cpython/cellobject.h similarity index 89% rename from Include/cellobject.h rename to Include/cpython/cellobject.h index 81bc784d36f3e..8dc7b8f4cf6f8 100644 --- a/Include/cellobject.h +++ b/Include/cpython/cellobject.h @@ -1,4 +1,5 @@ /* Cell object interface */ + #ifndef Py_LIMITED_API #ifndef Py_CELLOBJECT_H #define Py_CELLOBJECT_H @@ -8,7 +9,8 @@ extern "C" { typedef struct { PyObject_HEAD - PyObject *ob_ref; /* Content of the cell or NULL when empty */ + /* Content of the cell or NULL when empty */ + PyObject *ob_ref; } PyCellObject; PyAPI_DATA(PyTypeObject) PyCell_Type; diff --git a/Makefile.pre.in b/Makefile.pre.in index a5585c8de8ad0..32bbab068f702 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1127,7 +1127,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/boolobject.h \ $(srcdir)/Include/bytearrayobject.h \ $(srcdir)/Include/bytesobject.h \ - $(srcdir)/Include/cellobject.h \ $(srcdir)/Include/ceval.h \ $(srcdir)/Include/classobject.h \ $(srcdir)/Include/code.h \ @@ -1201,6 +1200,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/abstract.h \ $(srcdir)/Include/cpython/bytearrayobject.h \ $(srcdir)/Include/cpython/bytesobject.h \ + $(srcdir)/Include/cpython/cellobject.h \ $(srcdir)/Include/cpython/ceval.h \ $(srcdir)/Include/cpython/code.h \ $(srcdir)/Include/cpython/compile.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst index fc12e02b0c580..800f6e7f92741 100644 --- a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst +++ b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst @@ -1,3 +1,3 @@ -Move Include/funcobject.h header file to Include/cpython/funcobject.h. -C extensions should only include the main ```` header. -Patch by Victor Stinner. +Move ``cellobject.h`` and ``funcobject.h`` header files from ``Include/`` to +``Include/cpython/``. C extensions should only include the main ```` +header. Patch by Victor Stinner. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index dc216e34855df..877064e877deb 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -115,7 +115,6 @@ - @@ -126,6 +125,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 8eeb38871f327..b8841c90cc1b9 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -51,9 +51,6 @@ Include - - Include - Include @@ -372,6 +369,9 @@ Include\cpython + + Include\cpython + Include\cpython From webhook-mailer at python.org Fri Oct 15 00:29:01 2021 From: webhook-mailer at python.org (benjaminp) Date: Fri, 15 Oct 2021 04:29:01 -0000 Subject: [Python-checkins] closes bpo-45479: Degunkify Py_UniversalNewlineFgets. (GH-28965) Message-ID: https://github.com/python/cpython/commit/160c38df7fc7ba22dc687879c387bf643ffc3398 commit: 160c38df7fc7ba22dc687879c387bf643ffc3398 branch: main author: Benjamin Peterson committer: benjaminp date: 2021-10-14T21:28:52-07:00 summary: closes bpo-45479: Degunkify Py_UniversalNewlineFgets. (GH-28965) Remove dead variables and control flow. files: M Objects/fileobject.c diff --git a/Objects/fileobject.c b/Objects/fileobject.c index dc600c6d09a48..8eb62490c452b 100644 --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -239,13 +239,7 @@ _PyLong_FileDescriptor_Converter(PyObject *o, void *ptr) ** Py_UniversalNewlineFgets is an fgets variation that understands ** all of \r, \n and \r\n conventions. ** The stream should be opened in binary mode. -** If fobj is NULL the routine always does newline conversion, and -** it may peek one char ahead to gobble the second char in \r\n. -** If fobj is non-NULL it must be a PyFileObject. In this case there -** is no readahead but in stead a flag is used to skip a following -** \n on the next read. Also, if the file is open in binary mode -** the whole conversion is skipped. Finally, the routine keeps track of -** the different types of newlines seen. +** The fobj parameter exists solely for legacy reasons and must be NULL. ** Note that we need no error handling: fgets() treats error and eof ** identically. */ @@ -254,7 +248,6 @@ Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj) { char *p = buf; int c; - int newlinetypes = 0; int skipnextlf = 0; if (fobj) { @@ -262,24 +255,15 @@ Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj) return NULL; } FLOCKFILE(stream); - c = 'x'; /* Shut up gcc warning */ while (--n > 0 && (c = GETC(stream)) != EOF ) { - if (skipnextlf ) { + if (skipnextlf) { skipnextlf = 0; if (c == '\n') { /* Seeing a \n here with skipnextlf true ** means we saw a \r before. */ - newlinetypes |= NEWLINE_CRLF; c = GETC(stream); if (c == EOF) break; - } else { - /* - ** Note that c == EOF also brings us here, - ** so we're okay if the last char in the file - ** is a CR. - */ - newlinetypes |= NEWLINE_CR; } } if (c == '\r') { @@ -289,26 +273,15 @@ Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj) */ skipnextlf = 1; c = '\n'; - } else if ( c == '\n') { - newlinetypes |= NEWLINE_LF; } *p++ = c; if (c == '\n') break; } - /* if ( c == EOF && skipnextlf ) - newlinetypes |= NEWLINE_CR; */ FUNLOCKFILE(stream); *p = '\0'; - if ( skipnextlf ) { - /* If we have no file object we cannot save the - ** skipnextlf flag. We have to readahead, which - ** will cause a pause if we're reading from an - ** interactive stream, but that is very unlikely - ** unless we're doing something silly like - ** exec(open("/dev/tty").read()). - */ - c = GETC(stream); - if ( c != '\n' ) + if (skipnextlf) { + int c = GETC(stream); + if (c != '\n') ungetc(c, stream); } if (p == buf) From webhook-mailer at python.org Fri Oct 15 02:11:00 2021 From: webhook-mailer at python.org (benjaminp) Date: Fri, 15 Oct 2021 06:11:00 -0000 Subject: [Python-checkins] bpo-45479: Futher simplify Py_UniversalNewlineFgets. (GH-28967) Message-ID: https://github.com/python/cpython/commit/9ce9cfe595d64e3081e69de7296042cc54bccf18 commit: 9ce9cfe595d64e3081e69de7296042cc54bccf18 branch: main author: Benjamin Peterson committer: benjaminp date: 2021-10-14T23:10:52-07:00 summary: bpo-45479: Futher simplify Py_UniversalNewlineFgets. (GH-28967) Thank you to Eryk Sun for the suggestions in https://github.com/python/cpython/pull/28965#discussion_r729527143. files: M Objects/fileobject.c diff --git a/Objects/fileobject.c b/Objects/fileobject.c index 8eb62490c452b..8ca56a802b976 100644 --- a/Objects/fileobject.c +++ b/Objects/fileobject.c @@ -248,7 +248,6 @@ Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj) { char *p = buf; int c; - int skipnextlf = 0; if (fobj) { errno = ENXIO; /* What can you do... */ @@ -256,34 +255,21 @@ Py_UniversalNewlineFgets(char *buf, int n, FILE *stream, PyObject *fobj) } FLOCKFILE(stream); while (--n > 0 && (c = GETC(stream)) != EOF ) { - if (skipnextlf) { - skipnextlf = 0; - if (c == '\n') { - /* Seeing a \n here with skipnextlf true - ** means we saw a \r before. - */ - c = GETC(stream); - if (c == EOF) break; - } - } if (c == '\r') { - /* A \r is translated into a \n, and we skip - ** an adjacent \n, if any. We don't set the - ** newlinetypes flag until we've seen the next char. - */ - skipnextlf = 1; - c = '\n'; + // A \r is translated into a \n, and we skip an adjacent \n, if any. + c = GETC(stream); + if (c != '\n') { + ungetc(c, stream); + c = '\n'; + } } *p++ = c; - if (c == '\n') break; + if (c == '\n') { + break; + } } FUNLOCKFILE(stream); *p = '\0'; - if (skipnextlf) { - int c = GETC(stream); - if (c != '\n') - ungetc(c, stream); - } if (p == buf) return NULL; return buf; From webhook-mailer at python.org Fri Oct 15 03:46:38 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 07:46:38 -0000 Subject: [Python-checkins] bpo-35134: Move classobject.h to Include/cpython/ (GH-28968) Message-ID: https://github.com/python/cpython/commit/8e5de40f90476249e9a2e5ef135143b5c6a0b512 commit: 8e5de40f90476249e9a2e5ef135143b5c6a0b512 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T09:46:29+02:00 summary: bpo-35134: Move classobject.h to Include/cpython/ (GH-28968) Move classobject.h, context.h, genobject.h and longintrepr.h header files from Include/ to Include/cpython/. Remove redundant "#ifndef Py_LIMITED_API" in context.h. Remove explicit #include "longintrepr.h" in C files. It's not needed, Python.h already includes it. files: A Include/cpython/classobject.h A Include/cpython/context.h A Include/cpython/genobject.h A Include/cpython/longintrepr.h D Include/classobject.h D Include/context.h D Include/genobject.h D Include/longintrepr.h M Doc/whatsnew/3.11.rst M Include/Python.h M Makefile.pre.in M Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst M Modules/_decimal/_decimal.c M Objects/abstract.c M Objects/boolobject.c M Objects/longobject.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/marshal.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 734cf1572fcba..2e95228186579 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -572,7 +572,8 @@ Porting to Python 3.11 header provides functions like ``printf()`` and ``fopen()``. (Contributed by Victor Stinner in :issue:`45434`.) -* The non-limited API files ``cellobject.h`` and ``funcobject.h`` have been +* The non-limited API files ``cellobject.h``, ``classobject.h``, ``context.h``, + ``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to the ``Include/cpython`` directory. These files must not be included directly, as they are already included in ``Python.h``: :ref:`Include Files `. If they have been included directly, consider including diff --git a/Include/Python.h b/Include/Python.h index 89f60fe5c9f94..bc8d4f9b54889 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -46,7 +46,7 @@ #include "bytesobject.h" #include "unicodeobject.h" #include "longobject.h" -#include "longintrepr.h" +#include "cpython/longintrepr.h" #include "boolobject.h" #include "floatobject.h" #include "complexobject.h" @@ -61,7 +61,7 @@ #include "methodobject.h" #include "moduleobject.h" #include "cpython/funcobject.h" -#include "classobject.h" +#include "cpython/classobject.h" #include "fileobject.h" #include "pycapsule.h" #include "code.h" @@ -70,7 +70,8 @@ #include "sliceobject.h" #include "cpython/cellobject.h" #include "iterobject.h" -#include "genobject.h" +#include "pystate.h" +#include "cpython/genobject.h" #include "descrobject.h" #include "genericaliasobject.h" #include "warnings.h" @@ -83,8 +84,7 @@ #include "pyerrors.h" #include "cpython/initconfig.h" #include "pythread.h" -#include "pystate.h" -#include "context.h" +#include "cpython/context.h" #include "modsupport.h" #include "compile.h" #include "pythonrun.h" diff --git a/Include/classobject.h b/Include/cpython/classobject.h similarity index 96% rename from Include/classobject.h rename to Include/cpython/classobject.h index 1952f673b7d86..80df8842eb4f7 100644 --- a/Include/classobject.h +++ b/Include/cpython/classobject.h @@ -53,5 +53,5 @@ PyAPI_FUNC(PyObject *) PyInstanceMethod_Function(PyObject *); #ifdef __cplusplus } #endif -#endif /* !Py_CLASSOBJECT_H */ -#endif /* Py_LIMITED_API */ +#endif // !Py_CLASSOBJECT_H +#endif // !Py_LIMITED_API diff --git a/Include/context.h b/Include/cpython/context.h similarity index 99% rename from Include/context.h rename to Include/cpython/context.h index 4e5007089dd94..4db079f7633f4 100644 --- a/Include/context.h +++ b/Include/cpython/context.h @@ -1,12 +1,10 @@ +#ifndef Py_LIMITED_API #ifndef Py_CONTEXT_H #define Py_CONTEXT_H #ifdef __cplusplus extern "C" { #endif -#ifndef Py_LIMITED_API - - PyAPI_DATA(PyTypeObject) PyContext_Type; typedef struct _pycontextobject PyContext; @@ -73,9 +71,8 @@ PyAPI_FUNC(int) PyContextVar_Reset(PyObject *var, PyObject *token); PyAPI_FUNC(PyObject *) _PyContext_NewHamtForTests(void); -#endif /* !Py_LIMITED_API */ - #ifdef __cplusplus } #endif #endif /* !Py_CONTEXT_H */ +#endif /* !Py_LIMITED_API */ diff --git a/Include/genobject.h b/Include/cpython/genobject.h similarity index 93% rename from Include/genobject.h rename to Include/cpython/genobject.h index 55a8b34afd60e..8f87cf5fff757 100644 --- a/Include/genobject.h +++ b/Include/cpython/genobject.h @@ -1,4 +1,3 @@ - /* Generator object interface */ #ifndef Py_LIMITED_API @@ -8,8 +7,7 @@ extern "C" { #endif -#include "pystate.h" /* _PyErr_StackItem */ -#include "abstract.h" /* PySendResult */ +/* --- Generators --------------------------------------------------------- */ /* _PyGenObject_HEAD defines the initial segment of generator and coroutine objects. */ @@ -45,7 +43,9 @@ PyAPI_FUNC(int) _PyGen_FetchStopIterationValue(PyObject **); PyObject *_PyGen_yf(PyGenObject *); PyAPI_FUNC(void) _PyGen_Finalize(PyObject *self); -#ifndef Py_LIMITED_API + +/* --- PyCoroObject ------------------------------------------------------- */ + typedef struct { _PyGenObject_HEAD(cr) PyObject *cr_origin; @@ -59,7 +59,8 @@ PyObject *_PyCoro_GetAwaitableIter(PyObject *o); PyAPI_FUNC(PyObject *) PyCoro_New(PyFrameObject *, PyObject *name, PyObject *qualname); -/* Asynchronous Generators */ + +/* --- Asynchronous Generators -------------------------------------------- */ typedef struct { _PyGenObject_HEAD(ag) @@ -89,7 +90,6 @@ PyAPI_FUNC(PyObject *) PyAsyncGen_New(PyFrameObject *, PyObject *_PyAsyncGenValueWrapperNew(PyObject *); -#endif #undef _PyGenObject_HEAD diff --git a/Include/longintrepr.h b/Include/cpython/longintrepr.h similarity index 100% rename from Include/longintrepr.h rename to Include/cpython/longintrepr.h diff --git a/Makefile.pre.in b/Makefile.pre.in index 32bbab068f702..a81161e147d30 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1128,12 +1128,10 @@ PYTHON_HEADERS= \ $(srcdir)/Include/bytearrayobject.h \ $(srcdir)/Include/bytesobject.h \ $(srcdir)/Include/ceval.h \ - $(srcdir)/Include/classobject.h \ $(srcdir)/Include/code.h \ $(srcdir)/Include/codecs.h \ $(srcdir)/Include/compile.h \ $(srcdir)/Include/complexobject.h \ - $(srcdir)/Include/context.h \ $(srcdir)/Include/descrobject.h \ $(srcdir)/Include/dictobject.h \ $(srcdir)/Include/dynamic_annotations.h \ @@ -1144,13 +1142,11 @@ PYTHON_HEADERS= \ $(srcdir)/Include/fileutils.h \ $(srcdir)/Include/floatobject.h \ $(srcdir)/Include/frameobject.h \ - $(srcdir)/Include/genobject.h \ $(srcdir)/Include/import.h \ $(srcdir)/Include/interpreteridobject.h \ $(srcdir)/Include/intrcheck.h \ $(srcdir)/Include/iterobject.h \ $(srcdir)/Include/listobject.h \ - $(srcdir)/Include/longintrepr.h \ $(srcdir)/Include/longobject.h \ $(srcdir)/Include/marshal.h \ $(srcdir)/Include/memoryobject.h \ @@ -1202,18 +1198,22 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/bytesobject.h \ $(srcdir)/Include/cpython/cellobject.h \ $(srcdir)/Include/cpython/ceval.h \ + $(srcdir)/Include/cpython/classobject.h \ $(srcdir)/Include/cpython/code.h \ $(srcdir)/Include/cpython/compile.h \ + $(srcdir)/Include/cpython/context.h \ $(srcdir)/Include/cpython/dictobject.h \ $(srcdir)/Include/cpython/fileobject.h \ $(srcdir)/Include/cpython/fileutils.h \ $(srcdir)/Include/cpython/floatobject.h \ $(srcdir)/Include/cpython/frameobject.h \ $(srcdir)/Include/cpython/funcobject.h \ + $(srcdir)/Include/cpython/genobject.h \ $(srcdir)/Include/cpython/import.h \ $(srcdir)/Include/cpython/initconfig.h \ $(srcdir)/Include/cpython/interpreteridobject.h \ $(srcdir)/Include/cpython/listobject.h \ + $(srcdir)/Include/cpython/longintrepr.h \ $(srcdir)/Include/cpython/methodobject.h \ $(srcdir)/Include/cpython/object.h \ $(srcdir)/Include/cpython/objimpl.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst index 800f6e7f92741..4ab10884a5492 100644 --- a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst +++ b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst @@ -1,3 +1,4 @@ -Move ``cellobject.h`` and ``funcobject.h`` header files from ``Include/`` to +Move ``cellobject.h``, ``classobject.h``, ``context.h``, ``funcobject.h``, +``genobject.h`` and ``longintrepr.h`` header files from ``Include/`` to ``Include/cpython/``. C extensions should only include the main ```` header. Patch by Victor Stinner. diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index dd876f200365d..237edd5191fd9 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -28,7 +28,6 @@ #include #include "pycore_pystate.h" // _PyThreadState_GET() -#include "longintrepr.h" #include "complexobject.h" #include "mpdecimal.h" diff --git a/Objects/abstract.c b/Objects/abstract.c index 0d6cefd3eb861..6f7b94600e278 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -10,7 +10,6 @@ #include "pycore_unionobject.h" // _PyUnion_Check() #include #include // offsetof() -#include "longintrepr.h" diff --git a/Objects/boolobject.c b/Objects/boolobject.c index c72243a160b88..53f8192605797 100644 --- a/Objects/boolobject.c +++ b/Objects/boolobject.c @@ -2,7 +2,6 @@ #include "Python.h" #include "pycore_pyerrors.h" // _Py_FatalRefcountError() -#include "longintrepr.h" /* We define bool_repr to return "False" or "True" */ diff --git a/Objects/longobject.c b/Objects/longobject.c index 66e164974a92c..5325d1852bc02 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -8,7 +8,6 @@ #include "pycore_long.h" // __PyLong_GetSmallInt_internal() #include "pycore_object.h" // _PyObject_InitVar() #include "pycore_pystate.h" // _Py_IsMainInterpreter() -#include "longintrepr.h" #include #include diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 877064e877deb..0b0ff45621dcb 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -116,29 +116,31 @@ - - + + + + @@ -170,7 +172,6 @@ - @@ -224,7 +225,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index b8841c90cc1b9..17794fce88bc8 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -54,9 +54,6 @@ Include - - Include - Include @@ -69,9 +66,6 @@ Include - - Include - Include @@ -105,9 +99,6 @@ Include - - Include - Include @@ -120,9 +111,6 @@ Include - - Include - Include @@ -375,12 +363,18 @@ Include\cpython + + Include\cpython + Include\cpython Include + + Include\cpython + Include\cpython @@ -399,6 +393,9 @@ Include\cpython + + Include + Include @@ -456,6 +453,9 @@ Include\cpython + + Include + Include\cpython diff --git a/Python/marshal.c b/Python/marshal.c index e9ad566b71b10..51c77555d9ea9 100644 --- a/Python/marshal.c +++ b/Python/marshal.c @@ -13,7 +13,6 @@ #include "pycore_code.h" // _PyCode_New() #include "pycore_floatobject.h" // _PyFloat_Pack8() #include "pycore_hashtable.h" // _Py_hashtable_t -#include "longintrepr.h" #include "code.h" #include "marshal.h" // Py_MARSHAL_VERSION From webhook-mailer at python.org Fri Oct 15 05:39:06 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 15 Oct 2021 09:39:06 -0000 Subject: [Python-checkins] bpo-45428: Fix reading filenames from stdin in py_compile (GH-28848) Message-ID: https://github.com/python/cpython/commit/59a633d3e2071d65aa6638da5cf767a5c1310271 commit: 59a633d3e2071d65aa6638da5cf767a5c1310271 branch: main author: Graham Inggs committer: serhiy-storchaka date: 2021-10-15T12:38:55+03:00 summary: bpo-45428: Fix reading filenames from stdin in py_compile (GH-28848) Strip trailing '\n'. files: A Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst M Lib/py_compile.py diff --git a/Lib/py_compile.py b/Lib/py_compile.py index 0f9b59025cee3..388614e51b184 100644 --- a/Lib/py_compile.py +++ b/Lib/py_compile.py @@ -190,7 +190,7 @@ def main(): ) args = parser.parse_args() if args.filenames == ['-']: - filenames = sys.stdin.readlines() + filenames = [filename.rstrip('\n') for filename in sys.stdin.readlines()] else: filenames = args.filenames for filename in filenames: diff --git a/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst b/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst new file mode 100644 index 0000000000000..556eca43ed3c7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst @@ -0,0 +1 @@ +Fix a regression in py_compile when reading filenames from standard input. \ No newline at end of file From webhook-mailer at python.org Fri Oct 15 05:56:38 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 09:56:38 -0000 Subject: [Python-checkins] bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969) Message-ID: https://github.com/python/cpython/commit/063abd931f064a4b6b478b0b6e9aa13ee38d2cff commit: 063abd931f064a4b6b478b0b6e9aa13ee38d2cff branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T11:56:34+02:00 summary: bpo-35081: Move interpreteridobject.h to Include/internal/ (GH-28969) Move the interpreteridobject.h header file from Include/ to Include/internal/. It only provides private functions. files: A Include/internal/pycore_interpreteridobject.h A Misc/NEWS.d/next/C API/2021-10-15-09-29-59.bpo-35081.2teFD3.rst D Include/cpython/interpreteridobject.h D Include/interpreteridobject.h M Include/internal/pycore_pymem.h M Makefile.pre.in M Modules/_xxsubinterpretersmodule.c M Objects/interpreteridobject.c M Objects/object.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Tools/c-analyzer/cpython/_parser.py diff --git a/Include/cpython/interpreteridobject.h b/Include/internal/pycore_interpreteridobject.h similarity index 51% rename from Include/cpython/interpreteridobject.h rename to Include/internal/pycore_interpreteridobject.h index 5076584209b90..804831e76deae 100644 --- a/Include/cpython/interpreteridobject.h +++ b/Include/internal/pycore_interpreteridobject.h @@ -1,11 +1,22 @@ -#ifndef Py_CPYTHON_INTERPRETERIDOBJECT_H -# error "this header file must not be included directly" +/* Interpreter ID Object */ + +#ifndef Py_INTERNAL_INTERPRETERIDOBJECT_H +#define Py_INTERNAL_INTERPRETERIDOBJECT_H +#ifdef __cplusplus +extern "C" { #endif -/* Interpreter ID Object */ +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif PyAPI_DATA(PyTypeObject) _PyInterpreterID_Type; PyAPI_FUNC(PyObject *) _PyInterpreterID_New(int64_t); PyAPI_FUNC(PyObject *) _PyInterpreterState_GetIDObject(PyInterpreterState *); PyAPI_FUNC(PyInterpreterState *) _PyInterpreterID_LookUp(PyObject *); + +#ifdef __cplusplus +} +#endif +#endif // !Py_INTERNAL_INTERPRETERIDOBJECT_H diff --git a/Include/internal/pycore_pymem.h b/Include/internal/pycore_pymem.h index 30db3f2b6701e..d70deee710e30 100644 --- a/Include/internal/pycore_pymem.h +++ b/Include/internal/pycore_pymem.h @@ -103,4 +103,4 @@ void _PyObject_VirtualFree(void *, size_t size); #ifdef __cplusplus } #endif -#endif /* !Py_INTERNAL_PYMEM_H */ +#endif // !Py_INTERNAL_PYMEM_H diff --git a/Include/interpreteridobject.h b/Include/interpreteridobject.h deleted file mode 100644 index 8432632f339e9..0000000000000 --- a/Include/interpreteridobject.h +++ /dev/null @@ -1,17 +0,0 @@ -#ifndef Py_INTERPRETERIDOBJECT_H -#define Py_INTERPRETERIDOBJECT_H - -#ifdef __cplusplus -extern "C" { -#endif - -#ifndef Py_LIMITED_API -# define Py_CPYTHON_INTERPRETERIDOBJECT_H -# include "cpython/interpreteridobject.h" -# undef Py_CPYTHON_INTERPRETERIDOBJECT_H -#endif - -#ifdef __cplusplus -} -#endif -#endif /* !Py_INTERPRETERIDOBJECT_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index a81161e147d30..ccce52b36c42e 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1143,7 +1143,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/floatobject.h \ $(srcdir)/Include/frameobject.h \ $(srcdir)/Include/import.h \ - $(srcdir)/Include/interpreteridobject.h \ $(srcdir)/Include/intrcheck.h \ $(srcdir)/Include/iterobject.h \ $(srcdir)/Include/listobject.h \ @@ -1211,7 +1210,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/genobject.h \ $(srcdir)/Include/cpython/import.h \ $(srcdir)/Include/cpython/initconfig.h \ - $(srcdir)/Include/cpython/interpreteridobject.h \ $(srcdir)/Include/cpython/listobject.h \ $(srcdir)/Include/cpython/longintrepr.h \ $(srcdir)/Include/cpython/methodobject.h \ @@ -1260,6 +1258,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_import.h \ $(srcdir)/Include/internal/pycore_initconfig.h \ $(srcdir)/Include/internal/pycore_interp.h \ + $(srcdir)/Include/internal/pycore_interpreteridobject.h \ $(srcdir)/Include/internal/pycore_list.h \ $(srcdir)/Include/internal/pycore_long.h \ $(srcdir)/Include/internal/pycore_moduleobject.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-15-09-29-59.bpo-35081.2teFD3.rst b/Misc/NEWS.d/next/C API/2021-10-15-09-29-59.bpo-35081.2teFD3.rst new file mode 100644 index 0000000000000..79d4a9b9180d9 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-15-09-29-59.bpo-35081.2teFD3.rst @@ -0,0 +1,3 @@ +Move the ``interpreteridobject.h`` header file from ``Include/`` to +``Include/internal/``. It only provides private functions. Patch by Victor +Stinner. diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index 939114e60cae9..b5c0a63219114 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -6,7 +6,7 @@ #include "frameobject.h" #include "pycore_frame.h" #include "pycore_pystate.h" // _PyThreadState_GET() -#include "interpreteridobject.h" +#include "pycore_interpreteridobject.h" static char * diff --git a/Objects/interpreteridobject.c b/Objects/interpreteridobject.c index 46239100dcb7b..7b3e31beded59 100644 --- a/Objects/interpreteridobject.c +++ b/Objects/interpreteridobject.c @@ -3,7 +3,7 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_interp.h" // _PyInterpreterState_LookUpID() -#include "interpreteridobject.h" +#include "pycore_interpreteridobject.h" typedef struct interpid { diff --git a/Objects/object.c b/Objects/object.c index 589dd6647318d..5e719d45b60a6 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -16,7 +16,7 @@ #include "pycore_symtable.h" // PySTEntry_Type #include "pycore_unionobject.h" // _PyUnion_Type #include "frameobject.h" // PyFrame_Type -#include "interpreteridobject.h" // _PyInterpreterID_Type +#include "pycore_interpreteridobject.h" // _PyInterpreterID_Type #ifdef Py_LIMITED_API // Prevent recursive call _Py_IncRef() <=> Py_INCREF() diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 0b0ff45621dcb..32511d2a66511 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -138,7 +138,6 @@ - @@ -200,6 +199,7 @@ + @@ -221,7 +221,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 17794fce88bc8..4cc1092b33a49 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -333,9 +333,6 @@ Include - - Include - Modules @@ -456,9 +453,6 @@ Include - - Include\cpython - Include\cpython @@ -555,6 +549,9 @@ Include\internal + + Include\cpython + Include\internal diff --git a/Tools/c-analyzer/cpython/_parser.py b/Tools/c-analyzer/cpython/_parser.py index ef06a9fcb6903..8526b2af15a23 100644 --- a/Tools/c-analyzer/cpython/_parser.py +++ b/Tools/c-analyzer/cpython/_parser.py @@ -172,7 +172,6 @@ def clean_lines(text): Include/cpython/fileutils.h Py_CPYTHON_FILEUTILS_H 1 Include/cpython/frameobject.h Py_CPYTHON_FRAMEOBJECT_H 1 Include/cpython/import.h Py_CPYTHON_IMPORT_H 1 -Include/cpython/interpreteridobject.h Py_CPYTHON_INTERPRETERIDOBJECT_H 1 Include/cpython/listobject.h Py_CPYTHON_LISTOBJECT_H 1 Include/cpython/methodobject.h Py_CPYTHON_METHODOBJECT_H 1 Include/cpython/object.h Py_CPYTHON_OBJECT_H 1 From webhook-mailer at python.org Fri Oct 15 07:06:10 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 11:06:10 -0000 Subject: [Python-checkins] bpo-45434: Remove Include/eval.h header file (GH-28973) Message-ID: https://github.com/python/cpython/commit/105582e74c1817bc6a9d99bcb6540b34a3367292 commit: 105582e74c1817bc6a9d99bcb6540b34a3367292 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T13:06:05+02:00 summary: bpo-45434: Remove Include/eval.h header file (GH-28973) Move Include/eval.h content into Include/ceval.h and Include/cpython/ceval.h, and remove Include/eval.h. files: D Include/eval.h M Doc/faq/extending.rst M Doc/faq/windows.rst M Doc/whatsnew/3.11.rst M Include/Python.h M Include/ceval.h M Include/cpython/ceval.h M Makefile.pre.in M Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Doc/faq/extending.rst b/Doc/faq/extending.rst index 3379e41d9de07..fd32b097335e5 100644 --- a/Doc/faq/extending.rst +++ b/Doc/faq/extending.rst @@ -290,9 +290,6 @@ complete example using the GNU readline library (you may want to ignore #define PY_SSIZE_T_CLEAN #include - #include - #include - #include int main (int argc, char* argv[]) { diff --git a/Doc/faq/windows.rst b/Doc/faq/windows.rst index 0153a4f316ee8..6b95819c8ee85 100644 --- a/Doc/faq/windows.rst +++ b/Doc/faq/windows.rst @@ -212,7 +212,7 @@ Embedding the Python interpreter in a Windows app can be summarized as follows: .. code-block:: c - #include "python.h" + #include ... Py_Initialize(); // Initialize Python. initmyAppc(); // Initialize (import) the helper class. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 2e95228186579..994fb843fd377 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -573,11 +573,11 @@ Porting to Python 3.11 (Contributed by Victor Stinner in :issue:`45434`.) * The non-limited API files ``cellobject.h``, ``classobject.h``, ``context.h``, - ``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been - moved to the ``Include/cpython`` directory. These files must not be included - directly, as they are already included in ``Python.h``: :ref:`Include Files - `. If they have been included directly, consider including - ``Python.h`` instead. + ``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to + the ``Include/cpython`` directory. Moreover, the ``eval.h`` header file was + removed. These files must not be included directly, as they are already + included in ``Python.h``: :ref:`Include Files `. If they have + been included directly, consider including ``Python.h`` instead. (Contributed by Victor Stinner in :issue:`35134`.) Deprecated diff --git a/Include/Python.h b/Include/Python.h index bc8d4f9b54889..a2de514702f2f 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -96,7 +96,6 @@ #include "import.h" #include "abstract.h" #include "bltinmodule.h" -#include "eval.h" #include "cpython/pyctype.h" #include "pystrtod.h" #include "pystrcmp.h" diff --git a/Include/ceval.h b/Include/ceval.h index cf8c5b17be589..1b57f6ea20f6f 100644 --- a/Include/ceval.h +++ b/Include/ceval.h @@ -1,3 +1,5 @@ +/* Interface to random parts in ceval.c */ + #ifndef Py_CEVAL_H #define Py_CEVAL_H #ifdef __cplusplus @@ -5,7 +7,15 @@ extern "C" { #endif -/* Interface to random parts in ceval.c */ +PyAPI_FUNC(PyObject *) PyEval_EvalCode(PyObject *, PyObject *, PyObject *); + +PyAPI_FUNC(PyObject *) PyEval_EvalCodeEx(PyObject *co, + PyObject *globals, + PyObject *locals, + PyObject *const *args, int argc, + PyObject *const *kwds, int kwdc, + PyObject *const *defs, int defc, + PyObject *kwdefs, PyObject *closure); /* PyEval_CallObjectWithKeywords(), PyEval_CallObject(), PyEval_CallFunction * and PyEval_CallMethod are deprecated. Since they are officially part of the diff --git a/Include/cpython/ceval.h b/Include/cpython/ceval.h index 44b78f6d22312..caf64401307c0 100644 --- a/Include/cpython/ceval.h +++ b/Include/cpython/ceval.h @@ -2,6 +2,8 @@ # error "this header file must not be included directly" #endif +PyAPI_FUNC(PyObject *) _PyEval_CallTracing(PyObject *func, PyObject *args); + PyAPI_FUNC(void) PyEval_SetProfile(Py_tracefunc, PyObject *); PyAPI_DATA(int) _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg); PyAPI_FUNC(void) PyEval_SetTrace(Py_tracefunc, PyObject *); diff --git a/Include/eval.h b/Include/eval.h deleted file mode 100644 index eda28df8f6528..0000000000000 --- a/Include/eval.h +++ /dev/null @@ -1,27 +0,0 @@ - -/* Interface to execute compiled code */ - -#ifndef Py_EVAL_H -#define Py_EVAL_H -#ifdef __cplusplus -extern "C" { -#endif - -PyAPI_FUNC(PyObject *) PyEval_EvalCode(PyObject *, PyObject *, PyObject *); - -PyAPI_FUNC(PyObject *) PyEval_EvalCodeEx(PyObject *co, - PyObject *globals, - PyObject *locals, - PyObject *const *args, int argc, - PyObject *const *kwds, int kwdc, - PyObject *const *defs, int defc, - PyObject *kwdefs, PyObject *closure); - -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _PyEval_CallTracing(PyObject *func, PyObject *args); -#endif - -#ifdef __cplusplus -} -#endif -#endif /* !Py_EVAL_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index ccce52b36c42e..b79b71f59b145 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1137,7 +1137,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/dynamic_annotations.h \ $(srcdir)/Include/enumobject.h \ $(srcdir)/Include/errcode.h \ - $(srcdir)/Include/eval.h \ $(srcdir)/Include/fileobject.h \ $(srcdir)/Include/fileutils.h \ $(srcdir)/Include/floatobject.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst index 4ab10884a5492..d0d3ce6b34f6f 100644 --- a/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst +++ b/Misc/NEWS.d/next/C API/2021-10-15-00-11-51.bpo-35134.eX4zqy.rst @@ -1,4 +1,7 @@ -Move ``cellobject.h``, ``classobject.h``, ``context.h``, ``funcobject.h``, -``genobject.h`` and ``longintrepr.h`` header files from ``Include/`` to -``Include/cpython/``. C extensions should only include the main ```` -header. Patch by Victor Stinner. +The non-limited API files ``cellobject.h``, ``classobject.h``, ``context.h``, +``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to +the ``Include/cpython`` directory. Moreover, the ``eval.h`` header file was +removed. These files must not be included directly, as they are already +included in ``Python.h``: :ref:`Include Files `. If they have +been included directly, consider including ``Python.h`` instead. +Patch by Victor Stinner. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 32511d2a66511..357d0a7071031 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -166,7 +166,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 4cc1092b33a49..1a3ad884d4ac3 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -84,9 +84,6 @@ Include - - Include - Include From webhook-mailer at python.org Fri Oct 15 07:18:27 2021 From: webhook-mailer at python.org (pablogsal) Date: Fri, 15 Oct 2021 11:18:27 -0000 Subject: [Python-checkins] bpo-45445: Remove incorrectly commited test file (GH-28972) Message-ID: https://github.com/python/cpython/commit/79bc5e1dc6f87149240bded3654574b24168f1ac commit: 79bc5e1dc6f87149240bded3654574b24168f1ac branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-15T12:18:22+01:00 summary: bpo-45445: Remove incorrectly commited test file (GH-28972) files: D test_foo.py diff --git a/test_foo.py b/test_foo.py deleted file mode 100644 index a27be0fdb47a6..0000000000000 --- a/test_foo.py +++ /dev/null @@ -1,3 +0,0 @@ -def foo(a=3, *, c, d=2): - pass -foo() From webhook-mailer at python.org Fri Oct 15 07:43:52 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 11:43:52 -0000 Subject: [Python-checkins] bpo-44113: Move the What's New entry to Deprecate section (GH-28974) Message-ID: https://github.com/python/cpython/commit/a7f8dfd25a167ccfde9996c499fa38a2aba60022 commit: a7f8dfd25a167ccfde9996c499fa38a2aba60022 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T13:43:44+02:00 summary: bpo-44113: Move the What's New entry to Deprecate section (GH-28974) files: M Doc/whatsnew/3.11.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 994fb843fd377..8a3deaa0f6305 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -583,13 +583,6 @@ Porting to Python 3.11 Deprecated ---------- -Removed -------- - -* :c:func:`PyFrame_BlockSetup` and :c:func:`PyFrame_BlockPop` have been - removed. - (Contributed by Mark Shannon in :issue:`40222`.) - * Deprecate the following functions to configure the Python initialization: * :c:func:`PySys_AddWarnOptionUnicode` @@ -606,6 +599,13 @@ Removed ` instead (:pep:`587`). (Contributed by Victor Stinner in :issue:`44113`.) +Removed +------- + +* :c:func:`PyFrame_BlockSetup` and :c:func:`PyFrame_BlockPop` have been + removed. + (Contributed by Mark Shannon in :issue:`40222`.) + * Remove the following math macros using the ``errno`` variable: * ``Py_ADJUST_ERANGE1()`` From webhook-mailer at python.org Fri Oct 15 08:14:53 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 15 Oct 2021 12:14:53 -0000 Subject: [Python-checkins] bpo-45428: Fix reading filenames from stdin in py_compile (GH-28848) Message-ID: https://github.com/python/cpython/commit/2b6eb8149656541044884e76212495175e061a0a commit: 2b6eb8149656541044884e76212495175e061a0a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-15T05:14:35-07:00 summary: bpo-45428: Fix reading filenames from stdin in py_compile (GH-28848) Strip trailing '\n'. (cherry picked from commit 59a633d3e2071d65aa6638da5cf767a5c1310271) Co-authored-by: Graham Inggs files: A Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst M Lib/py_compile.py diff --git a/Lib/py_compile.py b/Lib/py_compile.py index 0f9b59025cee3..388614e51b184 100644 --- a/Lib/py_compile.py +++ b/Lib/py_compile.py @@ -190,7 +190,7 @@ def main(): ) args = parser.parse_args() if args.filenames == ['-']: - filenames = sys.stdin.readlines() + filenames = [filename.rstrip('\n') for filename in sys.stdin.readlines()] else: filenames = args.filenames for filename in filenames: diff --git a/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst b/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst new file mode 100644 index 0000000000000..556eca43ed3c7 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-14-18-04-17.bpo-45428.mM2War.rst @@ -0,0 +1 @@ +Fix a regression in py_compile when reading filenames from standard input. \ No newline at end of file From webhook-mailer at python.org Fri Oct 15 09:21:26 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 13:21:26 -0000 Subject: [Python-checkins] bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975) Message-ID: https://github.com/python/cpython/commit/354c35220d25a893e502014478f6739dad6897f3 commit: 354c35220d25a893e502014478f6739dad6897f3 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T15:21:21+02:00 summary: bpo-45482: Rename namespaceobject.h to pycore_namespace.h (GH-28975) Rename Include/namespaceobject.h to Include/internal/pycore_namespace.h. The _testmultiphase extension is now built with the Py_BUILD_CORE_MODULE macro defined to access _PyNamespace_Type. object.c: remove unused "pycore_context.h" include. files: A Include/internal/pycore_namespace.h D Include/namespaceobject.h M Include/Python.h M Makefile.pre.in M Modules/_testcapimodule.c M Modules/_testmultiphase.c M Modules/timemodule.c M Objects/namespaceobject.c M Objects/object.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/import.c M Python/sysmodule.c M setup.py diff --git a/Include/Python.h b/Include/Python.h index a2de514702f2f..c0a621ad44afd 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -77,7 +77,6 @@ #include "warnings.h" #include "weakrefobject.h" #include "structseq.h" -#include "namespaceobject.h" #include "cpython/picklebufobject.h" #include "cpython/pytime.h" #include "codecs.h" diff --git a/Include/internal/pycore_namespace.h b/Include/internal/pycore_namespace.h new file mode 100644 index 0000000000000..cb76f040693d1 --- /dev/null +++ b/Include/internal/pycore_namespace.h @@ -0,0 +1,20 @@ +// Simple namespace object interface + +#ifndef Py_INTERNAL_NAMESPACE_H +#define Py_INTERNAL_NAMESPACE_H +#ifdef __cplusplus +extern "C" { +#endif + +#ifndef Py_BUILD_CORE +# error "this header requires Py_BUILD_CORE define" +#endif + +PyAPI_DATA(PyTypeObject) _PyNamespace_Type; + +PyAPI_FUNC(PyObject *) _PyNamespace_New(PyObject *kwds); + +#ifdef __cplusplus +} +#endif +#endif // !Py_INTERNAL_NAMESPACE_H diff --git a/Include/namespaceobject.h b/Include/namespaceobject.h deleted file mode 100644 index 0c8d95c0f137c..0000000000000 --- a/Include/namespaceobject.h +++ /dev/null @@ -1,19 +0,0 @@ - -/* simple namespace object interface */ - -#ifndef NAMESPACEOBJECT_H -#define NAMESPACEOBJECT_H -#ifdef __cplusplus -extern "C" { -#endif - -#ifndef Py_LIMITED_API -PyAPI_DATA(PyTypeObject) _PyNamespace_Type; - -PyAPI_FUNC(PyObject *) _PyNamespace_New(PyObject *kwds); -#endif /* !Py_LIMITED_API */ - -#ifdef __cplusplus -} -#endif -#endif /* !NAMESPACEOBJECT_H */ diff --git a/Makefile.pre.in b/Makefile.pre.in index b79b71f59b145..9de51711ac4c5 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1151,7 +1151,6 @@ PYTHON_HEADERS= \ $(srcdir)/Include/methodobject.h \ $(srcdir)/Include/modsupport.h \ $(srcdir)/Include/moduleobject.h \ - $(srcdir)/Include/namespaceobject.h \ $(srcdir)/Include/object.h \ $(srcdir)/Include/objimpl.h \ $(srcdir)/Include/opcode.h \ @@ -1261,6 +1260,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/internal/pycore_list.h \ $(srcdir)/Include/internal/pycore_long.h \ $(srcdir)/Include/internal/pycore_moduleobject.h \ + $(srcdir)/Include/internal/pycore_namespace.h \ $(srcdir)/Include/internal/pycore_object.h \ $(srcdir)/Include/internal/pycore_pathconfig.h \ $(srcdir)/Include/internal/pycore_pyarena.h \ diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 7cbd2dc3b6a51..82cfc04c41096 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1167,8 +1167,8 @@ test_get_type_qualname(PyObject *self, PyObject *Py_UNUSED(ignored)) assert(strcmp(PyUnicode_AsUTF8(tp_qualname), "int") == 0); Py_DECREF(tp_qualname); - tp_qualname = PyType_GetQualName(&_PyNamespace_Type); - assert(strcmp(PyUnicode_AsUTF8(tp_qualname), "SimpleNamespace") == 0); + tp_qualname = PyType_GetQualName(&PyODict_Type); + assert(strcmp(PyUnicode_AsUTF8(tp_qualname), "OrderedDict") == 0); Py_DECREF(tp_qualname); PyObject *HeapTypeNameType = PyType_FromSpec(&HeapTypeNameType_Spec); diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index e0ed77d265cdc..2d25e16bd4d39 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -3,6 +3,7 @@ */ #include "Python.h" +#include "pycore_namespace.h" // _PyNamespace_New() /* State for testing module state access from methods */ diff --git a/Modules/timemodule.c b/Modules/timemodule.c index ca8d62d5cc938..0ef3b2ffaa898 100644 --- a/Modules/timemodule.c +++ b/Modules/timemodule.c @@ -2,6 +2,7 @@ #include "Python.h" #include "pycore_fileutils.h" // _Py_BEGIN_SUPPRESS_IPH +#include "pycore_namespace.h" // _PyNamespace_New() #include diff --git a/Objects/namespaceobject.c b/Objects/namespaceobject.c index fa37ed250d30a..7875e7cafecb6 100644 --- a/Objects/namespaceobject.c +++ b/Objects/namespaceobject.c @@ -1,6 +1,7 @@ // namespace object implementation #include "Python.h" +#include "pycore_namespace.h" // _PyNamespace_Type #include "structmember.h" // PyMemberDef diff --git a/Objects/object.c b/Objects/object.c index 5e719d45b60a6..25f5a2133d574 100644 --- a/Objects/object.c +++ b/Objects/object.c @@ -4,10 +4,10 @@ #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_ceval.h" // _Py_EnterRecursiveCall() -#include "pycore_context.h" -#include "pycore_dict.h" +#include "pycore_dict.h" // _PyObject_MakeDictFromInstanceAttributes() #include "pycore_floatobject.h" // _PyFloat_DebugMallocStats() #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() +#include "pycore_namespace.h" // _PyNamespace_Type #include "pycore_object.h" // _PyType_CheckConsistency() #include "pycore_pyerrors.h" // _PyErr_Occurred() #include "pycore_pylifecycle.h" // _PyTypes_InitSlotDefs() diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 357d0a7071031..015d783c89a83 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -202,6 +202,7 @@ + @@ -229,7 +230,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 1a3ad884d4ac3..94528b9261fa6 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -327,9 +327,6 @@ Include - - Include - Modules @@ -558,6 +555,9 @@ Include\internal + + Include + Include\internal diff --git a/Python/import.c b/Python/import.c index f2160928c4fa7..fe4686cd56b3b 100644 --- a/Python/import.c +++ b/Python/import.c @@ -3,19 +3,19 @@ #include "Python.h" #include "pycore_import.h" // _PyImport_BootstrapImp() -#include "pycore_initconfig.h" -#include "pycore_pyerrors.h" -#include "pycore_pyhash.h" +#include "pycore_initconfig.h" // _PyStatus_OK() +#include "pycore_interp.h" // _PyInterpreterState_ClearModules() +#include "pycore_namespace.h" // _PyNamespace_Type +#include "pycore_pyerrors.h" // _PyErr_SetString() +#include "pycore_pyhash.h" // _Py_KeyedHash() #include "pycore_pylifecycle.h" #include "pycore_pymem.h" // _PyMem_SetDefaultAllocator() -#include "pycore_interp.h" // _PyInterpreterState_ClearModules() #include "pycore_pystate.h" // _PyInterpreterState_GET() -#include "pycore_sysmodule.h" -#include "marshal.h" -#include "code.h" -#include "importdl.h" -#include "pydtrace.h" -#include +#include "pycore_sysmodule.h" // _PySys_Audit() +#include "marshal.h" // PyMarshal_ReadObjectFromString() +#include "importdl.h" // _PyImport_DynLoadFiletab +#include "pydtrace.h" // PyDTrace_IMPORT_FIND_LOAD_START_ENABLED() +#include // bool #ifdef HAVE_FCNTL_H #include diff --git a/Python/sysmodule.c b/Python/sysmodule.c index ae3cbf1954e09..ad9be714c021a 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -20,6 +20,7 @@ Data members: #include "pycore_code.h" // _Py_QuickenedCount #include "pycore_frame.h" // InterpreterFrame #include "pycore_initconfig.h" // _PyStatus_EXCEPTION() +#include "pycore_namespace.h" // _PyNamespace_New() #include "pycore_object.h" // _PyObject_IS_GC() #include "pycore_pathconfig.h" // _PyPathConfig_ComputeSysPath0() #include "pycore_pyerrors.h" // _PyErr_Fetch() diff --git a/setup.py b/setup.py index 56c06cbddaf21..24365ef3d927b 100644 --- a/setup.py +++ b/setup.py @@ -1043,7 +1043,8 @@ def detect_test_extensions(self): self.add(Extension('_testimportmultiple', ['_testimportmultiple.c'])) # Test multi-phase extension module init (PEP 489) - self.add(Extension('_testmultiphase', ['_testmultiphase.c'])) + self.add(Extension('_testmultiphase', ['_testmultiphase.c'], + extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) # Fuzz tests. self.add(Extension('_xxtestfuzz', From webhook-mailer at python.org Fri Oct 15 10:06:41 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 14:06:41 -0000 Subject: [Python-checkins] bpo-43760: Add PyThreadState_EnterTracing() (GH-28542) Message-ID: https://github.com/python/cpython/commit/547d26aa08aa5e4ec6e4f8a5587b30b39064a5ba commit: 547d26aa08aa5e4ec6e4f8a5587b30b39064a5ba branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T16:06:30+02:00 summary: bpo-43760: Add PyThreadState_EnterTracing() (GH-28542) Add PyThreadState_EnterTracing() and PyThreadState_LeaveTracing() functions to the limited C API to suspend and resume tracing and profiling. Add an unit test on the PyThreadState C API to _testcapi. Add also internal _PyThreadState_DisableTracing() and _PyThreadState_ResetTracing(). files: A Misc/NEWS.d/next/C API/2021-09-24-11-12-21.bpo-43760.Bfxd1-.rst M Doc/c-api/init.rst M Doc/whatsnew/3.11.rst M Include/cpython/pystate.h M Include/internal/pycore_pystate.h M Modules/_testcapimodule.c M Python/ceval.c M Python/pystate.c M Python/sysmodule.c diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 77036726b0a92..5e817268a2036 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1173,6 +1173,26 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. versionadded:: 3.9 +.. c:function:: void PyThreadState_EnterTracing(PyThreadState *tstate) + + Suspend tracing and profiling in the Python thread state *tstate*. + + Resume them using the:c:func:`PyThreadState_LeaveTracing` function. + + .. versionadded:: 3.11 + + +.. c:function:: void PyThreadState_LeaveTracing(PyThreadState *tstate) + + Resume tracing and profiling in the Python thread state *tstate* suspended + by the:c:func:`PyThreadState_EnterTracing` function. + + See also :c:func:`PyEval_SetTrace` and :c:func:`PyEval_SetProfile` + functions. + + .. versionadded:: 3.11 + + .. c:function:: PyInterpreterState* PyInterpreterState_Get(void) Get the current interpreter. @@ -1623,6 +1643,8 @@ Python-level trace functions in previous versions. profile function is called for all monitored events except :const:`PyTrace_LINE` :const:`PyTrace_OPCODE` and :const:`PyTrace_EXCEPTION`. + See also the :func:`sys.setprofile` function. + The caller must hold the :term:`GIL`. @@ -1635,6 +1657,8 @@ Python-level trace functions in previous versions. will not receive :const:`PyTrace_C_CALL`, :const:`PyTrace_C_EXCEPTION` or :const:`PyTrace_C_RETURN` as a value for the *what* parameter. + See also the :func:`sys.settrace` function. + The caller must hold the :term:`GIL`. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 8a3deaa0f6305..4704be5715426 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -476,6 +476,11 @@ New Features * Add a new :c:func:`PyType_GetQualName` function to get type's qualified name. (Contributed by Hai Shi in :issue:`42035`.) +* Add new :c:func:`PyThreadState_EnterTracing` and + :c:func:`PyThreadState_LeaveTracing` functions to the limited C API to + suspend and resume tracing and profiling. + (Contributed by Victor Stinner in :issue:`43760`.) + Porting to Python 3.11 ---------------------- diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index ca0de87ab0e5c..528d2a2998c99 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -185,6 +185,13 @@ PyAPI_FUNC(PyThreadState *) _PyThreadState_UncheckedGet(void); PyAPI_FUNC(PyObject *) _PyThreadState_GetDict(PyThreadState *tstate); +// Disable tracing and profiling. +PyAPI_FUNC(void) PyThreadState_EnterTracing(PyThreadState *tstate); + +// Reset tracing and profiling: enable them if a trace function or a profile +// function is set, otherwise disable them. +PyAPI_FUNC(void) PyThreadState_LeaveTracing(PyThreadState *tstate); + /* PyGILState */ /* Helper/diagnostic function - return 1 if the current thread diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index 1b74e5ffb0473..d63239736caba 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -125,7 +125,7 @@ static inline PyInterpreterState* _PyInterpreterState_GET(void) { } -/* Other */ +// PyThreadState functions PyAPI_FUNC(void) _PyThreadState_Init( PyThreadState *tstate); @@ -133,6 +133,23 @@ PyAPI_FUNC(void) _PyThreadState_DeleteExcept( _PyRuntimeState *runtime, PyThreadState *tstate); +static inline void +_PyThreadState_DisableTracing(PyThreadState *tstate) +{ + tstate->cframe->use_tracing = 0; +} + +static inline void +_PyThreadState_ResetTracing(PyThreadState *tstate) +{ + int use_tracing = (tstate->c_tracefunc != NULL + || tstate->c_profilefunc != NULL); + tstate->cframe->use_tracing = (use_tracing ? 255 : 0); +} + + +/* Other */ + PyAPI_FUNC(PyThreadState *) _PyThreadState_Swap( struct _gilstate_runtime_state *gilstate, PyThreadState *newts); diff --git a/Misc/NEWS.d/next/C API/2021-09-24-11-12-21.bpo-43760.Bfxd1-.rst b/Misc/NEWS.d/next/C API/2021-09-24-11-12-21.bpo-43760.Bfxd1-.rst new file mode 100644 index 0000000000000..16bea5ff8f57c --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-09-24-11-12-21.bpo-43760.Bfxd1-.rst @@ -0,0 +1,4 @@ +Add new :c:func:`PyThreadState_EnterTracing`, and +:c:func:`PyThreadState_LeaveTracing` functions to the limited C API to suspend +and resume tracing and profiling. +Patch by Victor Stinner. diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 82cfc04c41096..6857241999ea9 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -18,29 +18,32 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" -#include "datetime.h" -#include "marshal.h" +#include "frameobject.h" // PyFrame_Check() +#include "datetime.h" // PyDateTimeAPI +#include "marshal.h" // PyMarshal_WriteLongToFile #include "structmember.h" // PyMemberDef -#include +#include // FLT_MAX #include #ifdef MS_WINDOWS -# include /* struct timeval */ +# include // struct timeval #endif #ifdef HAVE_SYS_WAIT_H -#include /* For W_STOPCODE */ +#include // W_STOPCODE #endif #ifdef Py_BUILD_CORE # error "_testcapi must test the public Python C API, not CPython internal C API" #endif + +// Forward declarations static struct PyModuleDef _testcapimodule; static PyType_Spec HeapTypeNameType_Spec; - static PyObject *TestError; /* set to exception object in init */ + /* Raise TestError with test_name + ": " + msg, and return NULL. */ static PyObject * @@ -5674,6 +5677,57 @@ type_get_version(PyObject *self, PyObject *type) } +// Test PyThreadState C API +static PyObject * +test_tstate_capi(PyObject *self, PyObject *Py_UNUSED(args)) +{ + // PyThreadState_Get() + PyThreadState *tstate = PyThreadState_Get(); + assert(tstate != NULL); + + // PyThreadState_GET() + PyThreadState *tstate2 = PyThreadState_Get(); + assert(tstate2 == tstate); + + // private _PyThreadState_UncheckedGet() + PyThreadState *tstate3 = _PyThreadState_UncheckedGet(); + assert(tstate3 == tstate); + + // PyThreadState_EnterTracing(), PyThreadState_LeaveTracing() + PyThreadState_EnterTracing(tstate); + PyThreadState_LeaveTracing(tstate); + + // PyThreadState_GetDict(): no tstate argument + PyObject *dict = PyThreadState_GetDict(); + // PyThreadState_GetDict() API can return NULL if PyDict_New() fails, + // but it should not occur in practice. + assert(dict != NULL); + assert(PyDict_Check(dict)); + // dict is a borrowed reference + + // private _PyThreadState_GetDict() + PyObject *dict2 = _PyThreadState_GetDict(tstate); + assert(dict2 == dict); + // dict2 is a borrowed reference + + // PyThreadState_GetInterpreter() + PyInterpreterState *interp = PyThreadState_GetInterpreter(tstate); + assert(interp != NULL); + + // PyThreadState_GetFrame() + PyFrameObject*frame = PyThreadState_GetFrame(tstate); + assert(frame != NULL); + assert(PyFrame_Check(frame)); + Py_DECREF(frame); + + // PyThreadState_GetID() + uint64_t id = PyThreadState_GetID(tstate); + assert(id >= 1); + + Py_RETURN_NONE; +} + + static PyObject *test_buildvalue_issue38913(PyObject *, PyObject *); static PyObject *getargs_s_hash_int(PyObject *, PyObject *, PyObject*); @@ -5957,6 +6011,7 @@ static PyMethodDef TestMethods[] = { {"fatal_error", test_fatal_error, METH_VARARGS, PyDoc_STR("fatal_error(message, release_gil=False): call Py_FatalError(message)")}, {"type_get_version", type_get_version, METH_O, PyDoc_STR("type->tp_version_tag")}, + {"test_tstate_capi", test_tstate_capi, METH_NOARGS, NULL}, {NULL, NULL} /* sentinel */ }; diff --git a/Python/ceval.c b/Python/ceval.c index de71ae5da0f82..898687224e696 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -6187,7 +6187,7 @@ call_trace(Py_tracefunc func, PyObject *obj, if (tstate->tracing) return 0; tstate->tracing++; - tstate->cframe->use_tracing = 0; + _PyThreadState_DisableTracing(tstate); PyFrameObject *f = _PyFrame_GetFrameObject(frame); if (f == NULL) { return -1; @@ -6201,8 +6201,7 @@ call_trace(Py_tracefunc func, PyObject *obj, } result = func(obj, f, what, arg); f->f_lineno = 0; - tstate->cframe->use_tracing = ((tstate->c_tracefunc != NULL) - || (tstate->c_profilefunc != NULL)) ? 255 : 0; + _PyThreadState_ResetTracing(tstate); tstate->tracing--; return result; } @@ -6216,8 +6215,7 @@ _PyEval_CallTracing(PyObject *func, PyObject *args) PyObject *result; tstate->tracing = 0; - tstate->cframe->use_tracing = ((tstate->c_tracefunc != NULL) - || (tstate->c_profilefunc != NULL)) ? 255 : 0; + _PyThreadState_ResetTracing(tstate); result = PyObject_Call(func, args, NULL); tstate->tracing = save_tracing; tstate->cframe->use_tracing = save_use_tracing; @@ -6274,7 +6272,7 @@ _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_profilefunc = NULL; tstate->c_profileobj = NULL; /* Must make sure that tracing is not ignored if 'profileobj' is freed */ - tstate->cframe->use_tracing = tstate->c_tracefunc != NULL; + _PyThreadState_ResetTracing(tstate); Py_XDECREF(profileobj); Py_XINCREF(arg); @@ -6282,7 +6280,7 @@ _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_profilefunc = func; /* Flag that tracing or profiling is turned on */ - tstate->cframe->use_tracing = (func != NULL) || (tstate->c_tracefunc != NULL) ? 255 : 0; + _PyThreadState_ResetTracing(tstate); return 0; } @@ -6315,7 +6313,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_tracefunc = NULL; tstate->c_traceobj = NULL; /* Must make sure that profiling is not ignored if 'traceobj' is freed */ - tstate->cframe->use_tracing = (tstate->c_profilefunc != NULL) ? 255 : 0; + _PyThreadState_ResetTracing(tstate); Py_XDECREF(traceobj); Py_XINCREF(arg); @@ -6323,8 +6321,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_tracefunc = func; /* Flag that tracing or profiling is turned on */ - tstate->cframe->use_tracing = ((func != NULL) - || (tstate->c_profilefunc != NULL)) ? 255 : 0; + _PyThreadState_ResetTracing(tstate); return 0; } diff --git a/Python/pystate.c b/Python/pystate.c index ee9507c7dd80b..abd711ee2e796 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1201,6 +1201,22 @@ PyThreadState_SetAsyncExc(unsigned long id, PyObject *exc) } +void +PyThreadState_EnterTracing(PyThreadState *tstate) +{ + tstate->tracing++; + _PyThreadState_DisableTracing(tstate); +} + +void +PyThreadState_LeaveTracing(PyThreadState *tstate) +{ + tstate->tracing--; + _PyThreadState_ResetTracing(tstate); +} + + + /* Routines for advanced debuggers, requested by David Beazley. Don't use unless you know what you are doing! */ diff --git a/Python/sysmodule.c b/Python/sysmodule.c index ad9be714c021a..5e663c17c79b8 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -256,8 +256,7 @@ sys_audit_tstate(PyThreadState *ts, const char *event, } /* Disallow tracing in hooks unless explicitly enabled */ - ts->tracing++; - ts->cframe->use_tracing = 0; + PyThreadState_EnterTracing(ts); while ((hook = PyIter_Next(hooks)) != NULL) { _Py_IDENTIFIER(__cantrace__); PyObject *o; @@ -270,14 +269,12 @@ sys_audit_tstate(PyThreadState *ts, const char *event, break; } if (canTrace) { - ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc) ? 255 : 0; - ts->tracing--; + PyThreadState_LeaveTracing(ts); } PyObject* args[2] = {eventName, eventArgs}; o = _PyObject_FastCallTstate(ts, hook, args, 2); if (canTrace) { - ts->tracing++; - ts->cframe->use_tracing = 0; + PyThreadState_EnterTracing(ts); } if (!o) { break; @@ -285,8 +282,7 @@ sys_audit_tstate(PyThreadState *ts, const char *event, Py_DECREF(o); Py_CLEAR(hook); } - ts->cframe->use_tracing = (ts->c_tracefunc || ts->c_profilefunc) ? 255 : 0; - ts->tracing--; + PyThreadState_LeaveTracing(ts); if (_PyErr_Occurred(ts)) { goto exit; } From webhook-mailer at python.org Fri Oct 15 13:44:48 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 17:44:48 -0000 Subject: [Python-checkins] bpo-30459: Use (void) in macros setting variables (GH-28982) Message-ID: https://github.com/python/cpython/commit/51f8196d05f0e271358eee0f90fe044b20449fb5 commit: 51f8196d05f0e271358eee0f90fe044b20449fb5 branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T19:44:35+02:00 summary: bpo-30459: Use (void) in macros setting variables (GH-28982) Convert the result of macros setting variables to void to avoid risks of misusing them: * _PyGCHead_SET_NEXT() * asdl_seq_SET() * asdl_seq_SET_UNTYPED() files: M Include/internal/pycore_asdl.h M Include/internal/pycore_gc.h diff --git a/Include/internal/pycore_asdl.h b/Include/internal/pycore_asdl.h index c0b07c31810b9..2929e030872d9 100644 --- a/Include/internal/pycore_asdl.h +++ b/Include/internal/pycore_asdl.h @@ -91,7 +91,7 @@ asdl_ ## NAME ## _seq *_Py_asdl_ ## NAME ## _seq_new(Py_ssize_t size, PyArena *a (S)->typed_elements[_asdl_i] = (V); \ } while (0) #else -# define asdl_seq_SET(S, I, V) (S)->typed_elements[I] = (V) +# define asdl_seq_SET(S, I, V) ((void)((S)->typed_elements[I] = (V))) #endif #ifdef Py_DEBUG @@ -103,7 +103,7 @@ asdl_ ## NAME ## _seq *_Py_asdl_ ## NAME ## _seq_new(Py_ssize_t size, PyArena *a (S)->elements[_asdl_i] = (V); \ } while (0) #else -# define asdl_seq_SET_UNTYPED(S, I, V) (S)->elements[I] = (V) +# define asdl_seq_SET_UNTYPED(S, I, V) ((void)((S)->elements[I] = (V))) #endif #ifdef __cplusplus diff --git a/Include/internal/pycore_gc.h b/Include/internal/pycore_gc.h index 9db4a4716fa58..45e85b54c2ec2 100644 --- a/Include/internal/pycore_gc.h +++ b/Include/internal/pycore_gc.h @@ -43,7 +43,7 @@ typedef struct { // Lowest bit of _gc_next is used for flags only in GC. // But it is always 0 for normal code. #define _PyGCHead_NEXT(g) ((PyGC_Head*)(g)->_gc_next) -#define _PyGCHead_SET_NEXT(g, p) ((g)->_gc_next = (uintptr_t)(p)) +#define _PyGCHead_SET_NEXT(g, p) ((void)((g)->_gc_next = (uintptr_t)(p))) // Lowest two bits of _gc_prev is used for _PyGC_PREV_MASK_* flags. #define _PyGCHead_PREV(g) ((PyGC_Head*)((g)->_gc_prev & _PyGC_PREV_MASK)) @@ -56,7 +56,7 @@ typedef struct { #define _PyGCHead_FINALIZED(g) \ (((g)->_gc_prev & _PyGC_PREV_MASK_FINALIZED) != 0) #define _PyGCHead_SET_FINALIZED(g) \ - ((g)->_gc_prev |= _PyGC_PREV_MASK_FINALIZED) + ((void)((g)->_gc_prev |= _PyGC_PREV_MASK_FINALIZED)) #define _PyGC_FINALIZED(o) \ _PyGCHead_FINALIZED(_Py_AS_GC(o)) From webhook-mailer at python.org Fri Oct 15 13:45:39 2021 From: webhook-mailer at python.org (vstinner) Date: Fri, 15 Oct 2021 17:45:39 -0000 Subject: [Python-checkins] bpo-45440: Remove pymath.c fallbacks (GH-28977) Message-ID: https://github.com/python/cpython/commit/00ffc4513df7b89a168e88da4d1e3ac367f7682f commit: 00ffc4513df7b89a168e88da4d1e3ac367f7682f branch: main author: Victor Stinner committer: vstinner date: 2021-10-15T19:45:34+02:00 summary: bpo-45440: Remove pymath.c fallbacks (GH-28977) Remove fallbacks for missing round(), copysign() and hypot() in Python/pymath.c. Python now requires these functions to build. These fallbacks were needed on Visual Studio 2012 and older. They are no longer needed since Visual Stuido 2013. Python is now built with Visual Studio 2017 or newer since Python 3.6. files: M Doc/whatsnew/3.11.rst M Include/internal/pycore_pymath.h M Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst M PC/pyconfig.h M Python/pymath.c M configure M configure.ac M pyconfig.h.in diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 4704be5715426..9d7d3f19faae8 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -461,7 +461,8 @@ Build Changes (Contributed by Mike Gilbert in :issue:`45433`.) * Building Python now requires a C99 ```` header file providing - ``isinf()``, ``isnan()`` and ``isfinite()`` functions. + the following functions: ``copysign()``, ``hypot()``, ``isfinite()``, + ``isinf()``, ``isnan()``, ``round()``. (Contributed by Victor Stinner in :issue:`45440`.) C API Changes diff --git a/Include/internal/pycore_pymath.h b/Include/internal/pycore_pymath.h index 32743fc83f5c0..395b71452e40a 100644 --- a/Include/internal/pycore_pymath.h +++ b/Include/internal/pycore_pymath.h @@ -8,24 +8,6 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif - -// Python provides implementations for copysign(), round() and hypot() in -// Python/pymath.c just in case your math library doesn't provide the -// functions. -// -// Note: PC/pyconfig.h defines copysign as _copysign -#ifndef HAVE_COPYSIGN -extern double copysign(double, double); -#endif - -#ifndef HAVE_ROUND -extern double round(double); -#endif - -#ifndef HAVE_HYPOT -extern double hypot(double, double); -#endif - // Extra declarations #if !defined(_MSC_VER) && !defined(__STDC__) extern double fmod (double, double); diff --git a/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst b/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst index 1c7c413155f45..93f7f3139065f 100644 --- a/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst +++ b/Misc/NEWS.d/next/Build/2021-10-12-02-13-08.bpo-45440.-zYgDb.rst @@ -1,2 +1,4 @@ Building Python now requires a C99 ```` header file providing -``isinf()``, ``isnan()`` and ``isfinite()`` functions. Patch by Victor Stinner. +the following functions: ``copysign()``, ``hypot()``, ``isfinite()``, +``isinf()``, ``isnan()``, ``round()``. +Patch by Victor Stinner. diff --git a/PC/pyconfig.h b/PC/pyconfig.h index 75397772280dc..bb55ff4fe14c4 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -58,7 +58,6 @@ WIN32 is still required for the locale module. #include -#define HAVE_HYPOT #define HAVE_STRFTIME #define DONT_HAVE_SIG_ALARM #define DONT_HAVE_SIG_PAUSE @@ -348,14 +347,6 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */ /* Fairly standard from here! */ -/* Define to 1 if you have the `copysign' function. */ -#define HAVE_COPYSIGN 1 - -/* Define to 1 if you have the `round' function. */ -#if _MSC_VER >= 1800 -# define HAVE_ROUND 1 -#endif - /* Define if on AIX 3. System headers sometimes define this. We just want to avoid a redefinition error message. */ diff --git a/Python/pymath.c b/Python/pymath.c index b2681f2acc1f0..e7d0161ff94c2 100644 --- a/Python/pymath.c +++ b/Python/pymath.c @@ -17,51 +17,3 @@ void _Py_set_387controlword(unsigned short cw) { __asm__ __volatile__ ("fldcw %0" : : "m" (cw)); } #endif // HAVE_GCC_ASM_FOR_X87 - - -#ifndef HAVE_HYPOT -double hypot(double x, double y) -{ - double yx; - - x = fabs(x); - y = fabs(y); - if (x < y) { - double temp = x; - x = y; - y = temp; - } - if (x == 0.) - return 0.; - else { - yx = y/x; - return x*sqrt(1.+yx*yx); - } -} -#endif /* HAVE_HYPOT */ - -#ifndef HAVE_COPYSIGN -double -copysign(double x, double y) -{ - /* use atan2 to distinguish -0. from 0. */ - if (y > 0. || (y == 0. && atan2(y, -1.) > 0.)) { - return fabs(x); - } else { - return -fabs(x); - } -} -#endif /* HAVE_COPYSIGN */ - -#ifndef HAVE_ROUND -double -round(double x) -{ - double absx, y; - absx = fabs(x); - y = floor(absx); - if (absx - y >= 0.5) - y += 1.0; - return copysign(y, x); -} -#endif /* HAVE_ROUND */ diff --git a/configure b/configure index c56ff1dc5cd6a..81ee4282d9412 100755 --- a/configure +++ b/configure @@ -15066,7 +15066,7 @@ fi LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" -for ac_func in acosh asinh atanh copysign erf erfc expm1 finite gamma +for ac_func in acosh asinh atanh erf erfc expm1 finite gamma do : as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" @@ -15078,7 +15078,7 @@ _ACEOF fi done -for ac_func in hypot lgamma log1p log2 round tgamma +for ac_func in lgamma log1p log2 tgamma do : as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" diff --git a/configure.ac b/configure.ac index 1d0c753efac1a..ab3fc2839d4f8 100644 --- a/configure.ac +++ b/configure.ac @@ -4675,8 +4675,8 @@ fi LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" -AC_CHECK_FUNCS([acosh asinh atanh copysign erf erfc expm1 finite gamma]) -AC_CHECK_FUNCS([hypot lgamma log1p log2 round tgamma]) +AC_CHECK_FUNCS([acosh asinh atanh erf erfc expm1 finite gamma]) +AC_CHECK_FUNCS([lgamma log1p log2 tgamma]) # For multiprocessing module, check that sem_open # actually works. For FreeBSD versions <= 7.2, diff --git a/pyconfig.h.in b/pyconfig.h.in index 43a04e371a594..a426e8effddb9 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -160,9 +160,6 @@ /* Define to 1 if you have the header file. */ #undef HAVE_CONIO_H -/* Define to 1 if you have the `copysign' function. */ -#undef HAVE_COPYSIGN - /* Define to 1 if you have the `copy_file_range' function. */ #undef HAVE_COPY_FILE_RANGE @@ -538,9 +535,6 @@ /* Define this if you have le64toh() */ #undef HAVE_HTOLE64 -/* Define to 1 if you have the `hypot' function. */ -#undef HAVE_HYPOT - /* Define to 1 if you have the header file. */ #undef HAVE_IEEEFP_H @@ -872,9 +866,6 @@ /* Define if you have readline 4.0 */ #undef HAVE_RL_RESIZE_TERMINAL -/* Define to 1 if you have the `round' function. */ -#undef HAVE_ROUND - /* Define to 1 if you have the `rtpSpawn' function. */ #undef HAVE_RTPSPAWN From webhook-mailer at python.org Fri Oct 15 17:24:30 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 15 Oct 2021 21:24:30 -0000 Subject: [Python-checkins] bpo-45442: Add deactivate step to venv tutorial. (GH-28981) Message-ID: https://github.com/python/cpython/commit/11b2ae7f5bc0e7ebbfe944bb746a0b3dfcd7ff43 commit: 11b2ae7f5bc0e7ebbfe944bb746a0b3dfcd7ff43 branch: main author: srinivasan committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-15T14:24:25-07:00 summary: bpo-45442: Add deactivate step to venv tutorial. (GH-28981) @vsajip Sorry for the trouble?made a [fairly significant] git error in the previous PR. Have edited the patch as you had said in #28959. Automerge-Triggered-By: GH:vsajip files: M Doc/tutorial/venv.rst diff --git a/Doc/tutorial/venv.rst b/Doc/tutorial/venv.rst index 221c11ca74b4e..58ad31972f603 100644 --- a/Doc/tutorial/venv.rst +++ b/Doc/tutorial/venv.rst @@ -88,6 +88,11 @@ For example: '~/envs/tutorial-env/lib/python3.5/site-packages'] >>> +To deactivate a virtual environment, type:: + + deactivate + +into the terminal. Managing Packages with pip ========================== From webhook-mailer at python.org Sat Oct 16 08:55:23 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 16 Oct 2021 12:55:23 -0000 Subject: [Python-checkins] bpo-45463: Clarify that global statements allows multiple names (GH-28851) Message-ID: https://github.com/python/cpython/commit/4ecd119b007cb766b8bede2dc78b70d29cd932dd commit: 4ecd119b007cb766b8bede2dc78b70d29cd932dd branch: main author: Luca Chiodini committer: terryjreedy date: 2021-10-16T08:55:12-04:00 summary: bpo-45463: Clarify that global statements allows multiple names (GH-28851) The global statement allows specifying a list of identifiers (https://docs.python.org/3/reference/simple_stmts.html#the-global-statement). The "Execution model" chapter described the global statement as if it only allowed one single name. Pluralize "name" in the appropriate places. files: M Doc/reference/executionmodel.rst diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 55ac01b6a844d..5c85dd7052d67 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -119,14 +119,14 @@ is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. -If the :keyword:`global` statement occurs within a block, all uses of the name -specified in the statement refer to the binding of that name in the top-level +If the :keyword:`global` statement occurs within a block, all uses of the names +specified in the statement refer to the bindings of those names in the top-level namespace. Names are resolved in the top-level namespace by searching the global namespace, i.e. the namespace of the module containing the code block, and the builtins namespace, the namespace of the module :mod:`builtins`. The -global namespace is searched first. If the name is not found there, the +global namespace is searched first. If the names are not found there, the builtins namespace is searched. The :keyword:`!global` statement must precede -all uses of the name. +all uses of the listed names. The :keyword:`global` statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains From webhook-mailer at python.org Sat Oct 16 10:01:22 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 16 Oct 2021 14:01:22 -0000 Subject: [Python-checkins] [3.9]bpo-45463: Clarify that global statements allows multiple names (GH-28851) (GH-28990) Message-ID: https://github.com/python/cpython/commit/ac1b7a3319f268487c310ac7449703193f5eddad commit: ac1b7a3319f268487c310ac7449703193f5eddad branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: terryjreedy date: 2021-10-16T10:01:14-04:00 summary: [3.9]bpo-45463: Clarify that global statements allows multiple names (GH-28851) (GH-28990) The global statement allows specifying a list of identifiers (https://docs.python.org/3/reference/simple_stmts.htmlGH-the-global-statement). The "Execution model" chapter described the global statement as if it only allowed one single name. Pluralize "name" in the appropriate places. (cherry picked from commit 4ecd119b007cb766b8bede2dc78b70d29cd932dd) Co-authored-by: Luca Chiodini Co-authored-by: Luca Chiodini files: M Doc/reference/executionmodel.rst diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 55ac01b6a844d..5c85dd7052d67 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -119,14 +119,14 @@ is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. -If the :keyword:`global` statement occurs within a block, all uses of the name -specified in the statement refer to the binding of that name in the top-level +If the :keyword:`global` statement occurs within a block, all uses of the names +specified in the statement refer to the bindings of those names in the top-level namespace. Names are resolved in the top-level namespace by searching the global namespace, i.e. the namespace of the module containing the code block, and the builtins namespace, the namespace of the module :mod:`builtins`. The -global namespace is searched first. If the name is not found there, the +global namespace is searched first. If the names are not found there, the builtins namespace is searched. The :keyword:`!global` statement must precede -all uses of the name. +all uses of the listed names. The :keyword:`global` statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains From webhook-mailer at python.org Sat Oct 16 10:01:59 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 16 Oct 2021 14:01:59 -0000 Subject: [Python-checkins] [3.10]bpo-45463: Clarify that global statements allows multiple names (GH-28851) (GH-28989) Message-ID: https://github.com/python/cpython/commit/855d6247adb39d4e38b698b89e519587318abd80 commit: 855d6247adb39d4e38b698b89e519587318abd80 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: terryjreedy date: 2021-10-16T10:01:54-04:00 summary: [3.10]bpo-45463: Clarify that global statements allows multiple names (GH-28851) (GH-28989) The global statement allows specifying a list of identifiers (https://docs.python.org/3/reference/simple_stmts.htmlGH-the-global-statement). The "Execution model" chapter described the global statement as if it only allowed one single name. Pluralize "name" in the appropriate places. (cherry picked from commit 4ecd119b007cb766b8bede2dc78b70d29cd932dd) Co-authored-by: Luca Chiodini Co-authored-by: Luca Chiodini files: M Doc/reference/executionmodel.rst diff --git a/Doc/reference/executionmodel.rst b/Doc/reference/executionmodel.rst index 55ac01b6a844d..5c85dd7052d67 100644 --- a/Doc/reference/executionmodel.rst +++ b/Doc/reference/executionmodel.rst @@ -119,14 +119,14 @@ is subtle. Python lacks declarations and allows name binding operations to occur anywhere within a code block. The local variables of a code block can be determined by scanning the entire text of the block for name binding operations. -If the :keyword:`global` statement occurs within a block, all uses of the name -specified in the statement refer to the binding of that name in the top-level +If the :keyword:`global` statement occurs within a block, all uses of the names +specified in the statement refer to the bindings of those names in the top-level namespace. Names are resolved in the top-level namespace by searching the global namespace, i.e. the namespace of the module containing the code block, and the builtins namespace, the namespace of the module :mod:`builtins`. The -global namespace is searched first. If the name is not found there, the +global namespace is searched first. If the names are not found there, the builtins namespace is searched. The :keyword:`!global` statement must precede -all uses of the name. +all uses of the listed names. The :keyword:`global` statement has the same scope as a name binding operation in the same block. If the nearest enclosing scope for a free variable contains From webhook-mailer at python.org Sat Oct 16 11:13:07 2021 From: webhook-mailer at python.org (corona10) Date: Sat, 16 Oct 2021 15:13:07 -0000 Subject: [Python-checkins] bpo-45489: Update ForwardRef to support | operator. (GH-28991) Message-ID: https://github.com/python/cpython/commit/15ad52fbf607b6ccec44a38a8a32a5f1fad635ee commit: 15ad52fbf607b6ccec44a38a8a32a5f1fad635ee branch: main author: Dong-hee Na committer: corona10 date: 2021-10-17T00:12:58+09:00 summary: bpo-45489: Update ForwardRef to support | operator. (GH-28991) files: A Misc/NEWS.d/next/Library/2021-10-16-23-46-39.bpo-45489.QB0rhG.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 38397a07e5717..032fe91c7a840 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -2903,6 +2903,12 @@ def test_final_forward_ref(self): self.assertNotEqual(gth(Loop, globals())['attr'], Final[int]) self.assertNotEqual(gth(Loop, globals())['attr'], Final) + def test_or(self): + X = ForwardRef('X') + # __or__/__ror__ itself + self.assertEqual(X | "x", Union[X, "x"]) + self.assertEqual("x" | X, Union["x", X]) + class OverloadTests(BaseTestCase): diff --git a/Lib/typing.py b/Lib/typing.py index ada9adb0d32a5..78d973d2bba05 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -719,6 +719,12 @@ def __eq__(self, other): def __hash__(self): return hash(self.__forward_arg__) + def __or__(self, other): + return Union[self, other] + + def __ror__(self, other): + return Union[other, self] + def __repr__(self): return f'ForwardRef({self.__forward_arg__!r})' diff --git a/Misc/NEWS.d/next/Library/2021-10-16-23-46-39.bpo-45489.QB0rhG.rst b/Misc/NEWS.d/next/Library/2021-10-16-23-46-39.bpo-45489.QB0rhG.rst new file mode 100644 index 0000000000000..3921437d97ac5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-16-23-46-39.bpo-45489.QB0rhG.rst @@ -0,0 +1,2 @@ +Update :class:`~typing.ForwardRef` to support ``|`` operator. Patch by +Dong-hee Na. From webhook-mailer at python.org Sat Oct 16 11:16:59 2021 From: webhook-mailer at python.org (rhettinger) Date: Sat, 16 Oct 2021 15:16:59 -0000 Subject: [Python-checkins] bpo-42222: Remove deprecated support for non-integer values (GH-28983) Message-ID: https://github.com/python/cpython/commit/5afa0a411243210a30526c7459a0ccff5cb88494 commit: 5afa0a411243210a30526c7459a0ccff5cb88494 branch: main author: Raymond Hettinger committer: rhettinger date: 2021-10-16T10:16:53-05:00 summary: bpo-42222: Remove deprecated support for non-integer values (GH-28983) files: A Misc/NEWS.d/next/Library/2021-10-15-11-30-11.bpo-42222.hdHyac.rst M Doc/library/random.rst M Lib/random.py M Lib/test/test_random.py diff --git a/Doc/library/random.rst b/Doc/library/random.rst index e444f9573246a..0ac0fe72dc392 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -135,15 +135,10 @@ Functions for integers values. Formerly it used a style like ``int(random()*n)`` which could produce slightly uneven distributions. - .. deprecated:: 3.10 - The automatic conversion of non-integer types to equivalent integers is - deprecated. Currently ``randrange(10.0)`` is losslessly converted to - ``randrange(10)``. In the future, this will raise a :exc:`TypeError`. - - .. deprecated:: 3.10 - The exception raised for non-integral values such as ``randrange(10.5)`` - or ``randrange('10')`` will be changed from :exc:`ValueError` to - :exc:`TypeError`. + .. versionchanged:: 3.11 + Automatic conversion of non-integer types is no longer supported. + Calls such as ``randrange(10.0)`` and ``randrange(Fraction(10, 1))`` + now raise a :exc:`TypeError`. .. function:: randint(a, b) diff --git a/Lib/random.py b/Lib/random.py index 3569d58cc4243..92a71e14c480b 100644 --- a/Lib/random.py +++ b/Lib/random.py @@ -286,27 +286,17 @@ def randbytes(self, n): ## -------------------- integer methods ------------------- def randrange(self, start, stop=None, step=_ONE): - """Choose a random item from range(start, stop[, step]). + """Choose a random item from range(stop) or range(start, stop[, step]). - This fixes the problem with randint() which includes the - endpoint; in Python this is usually not what you want. + Roughly equivalent to ``choice(range(start, stop, step))`` + but supports arbitrarily large ranges and is optimized + for common cases. """ # This code is a bit messy to make it fast for the # common case while still doing adequate error checking. - try: - istart = _index(start) - except TypeError: - istart = int(start) - if istart != start: - _warn('randrange() will raise TypeError in the future', - DeprecationWarning, 2) - raise ValueError("non-integer arg 1 for randrange()") - _warn('non-integer arguments to randrange() have been deprecated ' - 'since Python 3.10 and will be removed in a subsequent ' - 'version', - DeprecationWarning, 2) + istart = _index(start) if stop is None: # We don't check for "step != 1" because it hasn't been # type checked and converted to an integer yet. @@ -316,37 +306,15 @@ def randrange(self, start, stop=None, step=_ONE): return self._randbelow(istart) raise ValueError("empty range for randrange()") - # stop argument supplied. - try: - istop = _index(stop) - except TypeError: - istop = int(stop) - if istop != stop: - _warn('randrange() will raise TypeError in the future', - DeprecationWarning, 2) - raise ValueError("non-integer stop for randrange()") - _warn('non-integer arguments to randrange() have been deprecated ' - 'since Python 3.10 and will be removed in a subsequent ' - 'version', - DeprecationWarning, 2) + # Stop argument supplied. + istop = _index(stop) width = istop - istart - try: - istep = _index(step) - except TypeError: - istep = int(step) - if istep != step: - _warn('randrange() will raise TypeError in the future', - DeprecationWarning, 2) - raise ValueError("non-integer step for randrange()") - _warn('non-integer arguments to randrange() have been deprecated ' - 'since Python 3.10 and will be removed in a subsequent ' - 'version', - DeprecationWarning, 2) + istep = _index(step) # Fast path. if istep == 1: if width > 0: return istart + self._randbelow(width) - raise ValueError("empty range for randrange() (%d, %d, %d)" % (istart, istop, width)) + raise ValueError(f"empty range in randrange({start}, {stop}, {step})") # Non-unit step argument supplied. if istep > 0: @@ -356,7 +324,7 @@ def randrange(self, start, stop=None, step=_ONE): else: raise ValueError("zero step for randrange()") if n <= 0: - raise ValueError("empty range for randrange()") + raise ValueError(f"empty range in randrange({start}, {stop}, {step})") return istart + istep * self._randbelow(n) def randint(self, a, b): diff --git a/Lib/test/test_random.py b/Lib/test/test_random.py index 448624b79be5d..3c5511d8c7f1a 100644 --- a/Lib/test/test_random.py +++ b/Lib/test/test_random.py @@ -481,50 +481,53 @@ def test_randrange_nonunit_step(self): self.assertEqual(rint, 0) def test_randrange_errors(self): - raises = partial(self.assertRaises, ValueError, self.gen.randrange) + raises_value_error = partial(self.assertRaises, ValueError, self.gen.randrange) + raises_type_error = partial(self.assertRaises, TypeError, self.gen.randrange) + # Empty range - raises(3, 3) - raises(-721) - raises(0, 100, -12) - # Non-integer start/stop - self.assertWarns(DeprecationWarning, raises, 3.14159) - self.assertWarns(DeprecationWarning, self.gen.randrange, 3.0) - self.assertWarns(DeprecationWarning, self.gen.randrange, Fraction(3, 1)) - self.assertWarns(DeprecationWarning, raises, '3') - self.assertWarns(DeprecationWarning, raises, 0, 2.71828) - self.assertWarns(DeprecationWarning, self.gen.randrange, 0, 2.0) - self.assertWarns(DeprecationWarning, self.gen.randrange, 0, Fraction(2, 1)) - self.assertWarns(DeprecationWarning, raises, 0, '2') - # Zero and non-integer step - raises(0, 42, 0) - self.assertWarns(DeprecationWarning, raises, 0, 42, 0.0) - self.assertWarns(DeprecationWarning, raises, 0, 0, 0.0) - self.assertWarns(DeprecationWarning, raises, 0, 42, 3.14159) - self.assertWarns(DeprecationWarning, self.gen.randrange, 0, 42, 3.0) - self.assertWarns(DeprecationWarning, self.gen.randrange, 0, 42, Fraction(3, 1)) - self.assertWarns(DeprecationWarning, raises, 0, 42, '3') - self.assertWarns(DeprecationWarning, self.gen.randrange, 0, 42, 1.0) - self.assertWarns(DeprecationWarning, raises, 0, 0, 1.0) + raises_value_error(3, 3) + raises_value_error(-721) + raises_value_error(0, 100, -12) + + # Zero step + raises_value_error(0, 42, 0) + + # Non-integer start/stop/step + raises_type_error(3.14159) + raises_type_error(3.0) + raises_type_error(Fraction(3, 1)) + raises_type_error('3') + raises_type_error(0, 2.71827) + raises_type_error(0, 2.0) + raises_type_error(0, Fraction(2, 1)) + raises_type_error(0, '2') + + # Non-integer step + raises_type_error(0, 42, 1.0) + raises_type_error(0, 0, 1.0) + raises_type_error(0, 42, 3.14159) + raises_type_error(0, 42, 3.0) + raises_type_error(0, 42, Fraction(3, 1)) + raises_type_error(0, 42, '3') + raises_type_error(0, 42, 1.0) + raises_type_error(0, 0, 1.0) def test_randrange_argument_handling(self): randrange = self.gen.randrange - with self.assertWarns(DeprecationWarning): + with self.assertRaises(TypeError): randrange(10.0, 20, 2) - with self.assertWarns(DeprecationWarning): + with self.assertRaises(TypeError): randrange(10, 20.0, 2) - with self.assertWarns(DeprecationWarning): + with self.assertRaises(TypeError): randrange(10, 20, 1.0) - with self.assertWarns(DeprecationWarning): + with self.assertRaises(TypeError): randrange(10, 20, 2.0) - with self.assertWarns(DeprecationWarning): - with self.assertRaises(ValueError): - randrange(10.5) - with self.assertWarns(DeprecationWarning): - with self.assertRaises(ValueError): - randrange(10, 20.5) - with self.assertWarns(DeprecationWarning): - with self.assertRaises(ValueError): - randrange(10, 20, 1.5) + with self.assertRaises(TypeError): + randrange(10.5) + with self.assertRaises(TypeError): + randrange(10, 20.5) + with self.assertRaises(TypeError): + randrange(10, 20, 1.5) def test_randrange_step(self): # bpo-42772: When stop is None, the step argument was being ignored. diff --git a/Misc/NEWS.d/next/Library/2021-10-15-11-30-11.bpo-42222.hdHyac.rst b/Misc/NEWS.d/next/Library/2021-10-15-11-30-11.bpo-42222.hdHyac.rst new file mode 100644 index 0000000000000..9b29fa44d7711 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-15-11-30-11.bpo-42222.hdHyac.rst @@ -0,0 +1 @@ +Removed deprecated support for float arguments in *randrange()*. From webhook-mailer at python.org Sat Oct 16 13:27:52 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Oct 2021 17:27:52 -0000 Subject: [Python-checkins] bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855) Message-ID: https://github.com/python/cpython/commit/fe0d9e22a52a10c4cbe52254b51f2d4e74d83568 commit: fe0d9e22a52a10c4cbe52254b51f2d4e74d83568 branch: main author: Pablo Galindo Salgado committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-16T10:27:43-07:00 summary: bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855) files: A Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst M Lib/test/test_traceback.py M Lib/traceback.py diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index 83d36e12c02d1..1c7db9d3d4737 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -55,6 +55,9 @@ def syntax_error_with_caret_non_ascii(self): def syntax_error_bad_indentation2(self): compile(" print(2)", "?", "exec") + def tokenizer_error_with_caret_range(self): + compile("blech ( ", "?", "exec") + def test_caret(self): err = self.get_exception_format(self.syntax_error_with_caret, SyntaxError) @@ -85,6 +88,13 @@ def test_caret(self): self.assertEqual(err[1].find("y"), err[2].find("^")) # in the right place self.assertEqual(err[2].count("^"), len("y for y in range(30)")) + err = self.get_exception_format(self.tokenizer_error_with_caret_range, + SyntaxError) + self.assertIn("^", err[2]) # third line has caret + self.assertEqual(err[2].count('\n'), 1) # and no additional newline + self.assertEqual(err[1].find("("), err[2].find("^")) # in the right place + self.assertEqual(err[2].count("^"), 1) + def test_nocaret(self): exc = SyntaxError("error", ("x.py", 23, None, "bad syntax")) err = traceback.format_exception_only(SyntaxError, exc) diff --git a/Lib/traceback.py b/Lib/traceback.py index 3cb8e5700de44..568f3ff28c29b 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -781,7 +781,7 @@ def _format_syntax_error(self, stype): if self.offset is not None: offset = self.offset - end_offset = self.end_offset if self.end_offset is not None else offset + end_offset = self.end_offset if self.end_offset not in {None, 0} else offset if offset == end_offset or end_offset == -1: end_offset = offset + 1 diff --git a/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst b/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst new file mode 100644 index 0000000000000..1d5a857e25435 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst @@ -0,0 +1,2 @@ +Fix the behaviour of :func:`traceback.print_exc` when displaying the caret +when the ``end_offset`` in the exception is set to 0. Patch by Pablo Galindo From webhook-mailer at python.org Sat Oct 16 13:51:13 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Oct 2021 17:51:13 -0000 Subject: [Python-checkins] bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855) Message-ID: https://github.com/python/cpython/commit/5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e commit: 5df35faf3611b459f7f32bfe9fd2ffa7fb2dc59e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-16T10:51:05-07:00 summary: bpo-45249: Fix caret location when end_offset is set to 0 (GH-28855) (cherry picked from commit fe0d9e22a52a10c4cbe52254b51f2d4e74d83568) Co-authored-by: Pablo Galindo Salgado files: A Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst M Lib/test/test_traceback.py M Lib/traceback.py diff --git a/Lib/test/test_traceback.py b/Lib/test/test_traceback.py index fabe5d01a94db..ba03ce46294c5 100644 --- a/Lib/test/test_traceback.py +++ b/Lib/test/test_traceback.py @@ -51,6 +51,9 @@ def syntax_error_with_caret_non_ascii(self): def syntax_error_bad_indentation2(self): compile(" print(2)", "?", "exec") + def tokenizer_error_with_caret_range(self): + compile("blech ( ", "?", "exec") + def test_caret(self): err = self.get_exception_format(self.syntax_error_with_caret, SyntaxError) @@ -81,6 +84,13 @@ def test_caret(self): self.assertEqual(err[1].find("y"), err[2].find("^")) # in the right place self.assertEqual(err[2].count("^"), len("y for y in range(30)")) + err = self.get_exception_format(self.tokenizer_error_with_caret_range, + SyntaxError) + self.assertIn("^", err[2]) # third line has caret + self.assertEqual(err[2].count('\n'), 1) # and no additional newline + self.assertEqual(err[1].find("("), err[2].find("^")) # in the right place + self.assertEqual(err[2].count("^"), 1) + def test_nocaret(self): exc = SyntaxError("error", ("x.py", 23, None, "bad syntax")) err = traceback.format_exception_only(SyntaxError, exc) diff --git a/Lib/traceback.py b/Lib/traceback.py index c7947f118beaa..901b99476aaf8 100644 --- a/Lib/traceback.py +++ b/Lib/traceback.py @@ -633,7 +633,7 @@ def _format_syntax_error(self, stype): if self.offset is not None: offset = self.offset - end_offset = self.end_offset if self.end_offset is not None else offset + end_offset = self.end_offset if self.end_offset not in {None, 0} else offset if offset == end_offset or end_offset == -1: end_offset = offset + 1 diff --git a/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst b/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst new file mode 100644 index 0000000000000..1d5a857e25435 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-10-16-14-33.bpo-45249.xqLliz.rst @@ -0,0 +1,2 @@ +Fix the behaviour of :func:`traceback.print_exc` when displaying the caret +when the ``end_offset`` in the exception is set to 0. Patch by Pablo Galindo From webhook-mailer at python.org Sat Oct 16 15:16:17 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Sat, 16 Oct 2021 19:16:17 -0000 Subject: [Python-checkins] bpo-45020: Default to using frozen modules unless running from source tree. (gh-28940) Message-ID: https://github.com/python/cpython/commit/b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b commit: b9cdd0fb9c463c2503a4d854bb6529a9db58fe1b branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-16T13:16:08-06:00 summary: bpo-45020: Default to using frozen modules unless running from source tree. (gh-28940) The default was "off". Switching it to "on" means users get the benefit of frozen stdlib modules without having to do anything. There's a special-case for running-in-source-tree, so contributors don't get surprised when their stdlib changes don't get used. https://bugs.python.org/issue45020 files: M Doc/using/cmdline.rst M Include/internal/pycore_fileutils.h M Lib/site.py M Lib/test/test_embed.py M Misc/NEWS.d/3.11.0a1.rst M Python/fileutils.c M Python/initconfig.c diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 23a645a036aed..d341ea8bb43c8 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -483,7 +483,8 @@ Miscellaneous options * ``-X frozen_modules`` determines whether or not frozen modules are ignored by the import machinery. A value of "on" means they get imported and "off" means they are ignored. The default is "on" - for non-debug builds (the normal case) and "off" for debug builds. + if this is an installed Python (the normal case). If it's under + development (running from the source tree) then the default is "off". Note that the "importlib_bootstrap" and "importlib_bootstrap_external" frozen modules are always used, even if this flag is set to "off". diff --git a/Include/internal/pycore_fileutils.h b/Include/internal/pycore_fileutils.h index 3464477bce575..ab436ae9b007a 100644 --- a/Include/internal/pycore_fileutils.h +++ b/Include/internal/pycore_fileutils.h @@ -79,6 +79,7 @@ extern wchar_t * _Py_join_relfile(const wchar_t *dirname, extern int _Py_add_relfile(wchar_t *dirname, const wchar_t *relfile, size_t bufsize); +extern size_t _Py_find_basename(const wchar_t *filename); // Macros to protect CRT calls against instant termination when passed an // invalid parameter (bpo-23524). IPH stands for Invalid Parameter Handler. diff --git a/Lib/site.py b/Lib/site.py index 939893eb5ee93..e129f3b4851f3 100644 --- a/Lib/site.py +++ b/Lib/site.py @@ -418,8 +418,10 @@ def setcopyright(): files, dirs = [], [] # Not all modules are required to have a __file__ attribute. See # PEP 420 for more details. - if hasattr(os, '__file__'): + here = getattr(sys, '_stdlib_dir', None) + if not here and hasattr(os, '__file__'): here = os.path.dirname(os.__file__) + if here: files.extend(["LICENSE.txt", "LICENSE"]) dirs.extend([os.path.join(here, os.pardir), here, os.curdir]) builtins.license = _sitebuiltins._Printer( diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 4cbb4c2c1ce36..4b4396efb5cad 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -53,12 +53,13 @@ def remove_python_envvars(): class EmbeddingTestsMixin: def setUp(self): exename = "_testembed" + builddir = os.path.dirname(sys.executable) if MS_WINDOWS: ext = ("_d" if debug_build(sys.executable) else "") + ".exe" exename += ext - exepath = os.path.dirname(sys.executable) + exepath = builddir else: - exepath = os.path.join(support.REPO_ROOT, "Programs") + exepath = os.path.join(builddir, 'Programs') self.test_exe = exe = os.path.join(exepath, exename) if not os.path.exists(exe): self.skipTest("%r doesn't exist" % exe) @@ -434,7 +435,7 @@ class InitConfigTests(EmbeddingTestsMixin, unittest.TestCase): 'pathconfig_warnings': 1, '_init_main': 1, '_isolated_interpreter': 0, - 'use_frozen_modules': 0, + 'use_frozen_modules': 1, } if MS_WINDOWS: CONFIG_COMPAT.update({ @@ -1146,6 +1147,7 @@ def test_init_setpath(self): # The current getpath.c doesn't determine the stdlib dir # in this case. 'stdlib_dir': '', + 'use_frozen_modules': -1, } self.default_program_name(config) env = {'TESTPATH': os.path.pathsep.join(paths)} @@ -1169,6 +1171,7 @@ def test_init_setpath_config(self): # The current getpath.c doesn't determine the stdlib dir # in this case. 'stdlib_dir': '', + 'use_frozen_modules': -1, # overridden by PyConfig 'program_name': 'conf_program_name', 'base_executable': 'conf_executable', @@ -1265,6 +1268,8 @@ def test_init_setpythonhome(self): 'stdlib_dir': stdlib, } self.default_program_name(config) + if not config['executable']: + config['use_frozen_modules'] = -1 env = {'TESTHOME': home, 'PYTHONPATH': paths_str} self.check_all_configs("test_init_setpythonhome", config, api=API_COMPAT, env=env) @@ -1303,6 +1308,7 @@ def test_init_pybuilddir(self): # The current getpath.c doesn't determine the stdlib dir # in this case. 'stdlib_dir': None, + 'use_frozen_modules': -1, } env = self.copy_paths_by_env(config) self.check_all_configs("test_init_compat_config", config, @@ -1361,6 +1367,7 @@ def test_init_pyvenv_cfg(self): config['base_prefix'] = pyvenv_home config['prefix'] = pyvenv_home config['stdlib_dir'] = os.path.join(pyvenv_home, 'lib') + config['use_frozen_modules'] = 1 ver = sys.version_info dll = f'python{ver.major}' @@ -1373,6 +1380,7 @@ def test_init_pyvenv_cfg(self): # The current getpath.c doesn't determine the stdlib dir # in this case. config['stdlib_dir'] = None + config['use_frozen_modules'] = -1 env = self.copy_paths_by_env(config) self.check_all_configs("test_init_compat_config", config, diff --git a/Misc/NEWS.d/3.11.0a1.rst b/Misc/NEWS.d/3.11.0a1.rst index e3d2acc499968..a64a3e74ccb41 100644 --- a/Misc/NEWS.d/3.11.0a1.rst +++ b/Misc/NEWS.d/3.11.0a1.rst @@ -262,7 +262,7 @@ Compiler now removes trailing unused constants from co_consts. Add a new command line option, "-X frozen_modules=[on|off]" to opt out of (or into) using optional frozen modules. This defaults to "on" (or "off" if -it's a debug build). +it's running out of the source tree). .. diff --git a/Python/fileutils.c b/Python/fileutils.c index 173d34dd23f18..3d8f3a4f16326 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -2169,6 +2169,18 @@ _Py_add_relfile(wchar_t *dirname, const wchar_t *relfile, size_t bufsize) } +size_t +_Py_find_basename(const wchar_t *filename) +{ + for (size_t i = wcslen(filename); i > 0; --i) { + if (filename[i] == SEP) { + return i + 1; + } + } + return 0; +} + + /* Get the current directory. buflen is the buffer size in wide characters including the null character. Decode the path from the locale encoding. diff --git a/Python/initconfig.c b/Python/initconfig.c index b0d54b0472fb0..c916e2f7c0714 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -739,6 +739,7 @@ _PyConfig_InitCompatConfig(PyConfig *config) #ifdef MS_WINDOWS config->legacy_windows_stdio = -1; #endif + config->use_frozen_modules = -1; } @@ -2090,6 +2091,44 @@ config_init_fs_encoding(PyConfig *config, const PyPreConfig *preconfig) } +/* Determine if the current build is a "development" build (e.g. running + out of the source tree) or not. + + A return value of -1 indicates that we do not know. + */ +static int +is_dev_env(PyConfig *config) +{ + // This should only ever get called early in runtime initialization, + // before the global path config is written. Otherwise we would + // use Py_GetProgramFullPath() and _Py_GetStdlibDir(). + assert(config != NULL); + + const wchar_t *executable = config->executable; + const wchar_t *stdlib = config->stdlib_dir; + if (executable == NULL || *executable == L'\0' || + stdlib == NULL || *stdlib == L'\0') { + // _PyPathConfig_Calculate() hasn't run yet. + return -1; + } + size_t len = _Py_find_basename(executable); + if (wcscmp(executable + len, L"python") != 0 && + wcscmp(executable + len, L"python.exe") != 0) { + return 0; + } + /* If dirname() is the same for both then it is a dev build. */ + if (len != _Py_find_basename(stdlib)) { + return 0; + } + // We do not bother normalizing the two filenames first since + // for config_init_import() is does the right thing as-is. + if (wcsncmp(stdlib, executable, len) != 0) { + return 0; + } + return 1; +} + + static PyStatus config_init_import(PyConfig *config, int compute_path_config) { @@ -2101,25 +2140,28 @@ config_init_import(PyConfig *config, int compute_path_config) } /* -X frozen_modules=[on|off] */ - const wchar_t *value = config_get_xoption_value(config, L"frozen_modules"); - if (value == NULL) { - // For now we always default to "off". - // In the near future we will be factoring in PGO and in-development. - config->use_frozen_modules = 0; - } - else if (wcscmp(value, L"on") == 0) { - config->use_frozen_modules = 1; - } - else if (wcscmp(value, L"off") == 0) { - config->use_frozen_modules = 0; - } - else if (wcslen(value) == 0) { - // "-X frozen_modules" and "-X frozen_modules=" both imply "on". - config->use_frozen_modules = 1; - } - else { - return PyStatus_Error("bad value for option -X frozen_modules " - "(expected \"on\" or \"off\")"); + if (config->use_frozen_modules < 0) { + const wchar_t *value = config_get_xoption_value(config, L"frozen_modules"); + if (value == NULL) { + int isdev = is_dev_env(config); + if (isdev >= 0) { + config->use_frozen_modules = !isdev; + } + } + else if (wcscmp(value, L"on") == 0) { + config->use_frozen_modules = 1; + } + else if (wcscmp(value, L"off") == 0) { + config->use_frozen_modules = 0; + } + else if (wcslen(value) == 0) { + // "-X frozen_modules" and "-X frozen_modules=" both imply "on". + config->use_frozen_modules = 1; + } + else { + return PyStatus_Error("bad value for option -X frozen_modules " + "(expected \"on\" or \"off\")"); + } } return _PyStatus_OK(); From webhook-mailer at python.org Sat Oct 16 18:44:08 2021 From: webhook-mailer at python.org (terryjreedy) Date: Sat, 16 Oct 2021 22:44:08 -0000 Subject: [Python-checkins] bpo-45495: Add 'case' and 'match' to IDLE completions list. (GH-29000) Message-ID: https://github.com/python/cpython/commit/42ac06dcd234bdda989dcfe854ac5173337024c9 commit: 42ac06dcd234bdda989dcfe854ac5173337024c9 branch: main author: Terry Jan Reedy committer: terryjreedy date: 2021-10-16T18:44:00-04:00 summary: bpo-45495: Add 'case' and 'match' to IDLE completions list. (GH-29000) Since the keyword list is frozen, only compute it once per session. The colorizer already handles context keywords. files: A Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst M Lib/idlelib/autocomplete.py M Lib/idlelib/idle_test/test_autocomplete.py diff --git a/Lib/idlelib/autocomplete.py b/Lib/idlelib/autocomplete.py index bb7ee035c4fef..032d31225315f 100644 --- a/Lib/idlelib/autocomplete.py +++ b/Lib/idlelib/autocomplete.py @@ -9,6 +9,12 @@ import string import sys +# Modified keyword list is used in fetch_completions. +completion_kwds = [s for s in keyword.kwlist + if s not in {'True', 'False', 'None'}] # In builtins. +completion_kwds.extend(('match', 'case')) # Context keywords. +completion_kwds.sort() + # Two types of completions; defined here for autocomplete_w import below. ATTRS, FILES = 0, 1 from idlelib import autocomplete_w @@ -177,9 +183,7 @@ def fetch_completions(self, what, mode): namespace = {**__main__.__builtins__.__dict__, **__main__.__dict__} bigl = eval("dir()", namespace) - kwds = (s for s in keyword.kwlist - if s not in {'True', 'False', 'None'}) - bigl.extend(kwds) + bigl.extend(completion_kwds) bigl.sort() if "__all__" in bigl: smalll = sorted(eval("__all__", namespace)) diff --git a/Lib/idlelib/idle_test/test_autocomplete.py b/Lib/idlelib/idle_test/test_autocomplete.py index 642bb5db64dc3..a811363c18d04 100644 --- a/Lib/idlelib/idle_test/test_autocomplete.py +++ b/Lib/idlelib/idle_test/test_autocomplete.py @@ -218,6 +218,11 @@ def make_acw(): return self.dummy_acw() self.assertTrue(acp.open_completions(ac.TAB)) self.text.delete('1.0', 'end') + def test_completion_kwds(self): + self.assertIn('and', ac.completion_kwds) + self.assertIn('case', ac.completion_kwds) + self.assertNotIn('None', ac.completion_kwds) + def test_fetch_completions(self): # Test that fetch_completions returns 2 lists: # For attribute completion, a large list containing all variables, and diff --git a/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst b/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst new file mode 100644 index 0000000000000..3868f8d136a04 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst @@ -0,0 +1 @@ +Add context keywords 'case' and 'match' to completions list. From webhook-mailer at python.org Sat Oct 16 19:14:20 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 16 Oct 2021 23:14:20 -0000 Subject: [Python-checkins] [3.10] bpo-45495: Add 'case' and 'match' to IDLE completions list. (GH-29000) (GH-29001) Message-ID: https://github.com/python/cpython/commit/a29470307308f64af9c55263cdd6e1984ba89925 commit: a29470307308f64af9c55263cdd6e1984ba89925 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-16T16:14:11-07:00 summary: [3.10] bpo-45495: Add 'case' and 'match' to IDLE completions list. (GH-29000) (GH-29001) Since the keyword list is frozen, only compute it once per session. The colorizer already handles context keywords. (cherry picked from commit 42ac06dcd234bdda989dcfe854ac5173337024c9) Co-authored-by: Terry Jan Reedy Automerge-Triggered-By: GH:terryjreedy files: A Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst M Lib/idlelib/autocomplete.py M Lib/idlelib/idle_test/test_autocomplete.py diff --git a/Lib/idlelib/autocomplete.py b/Lib/idlelib/autocomplete.py index bb7ee035c4fef..032d31225315f 100644 --- a/Lib/idlelib/autocomplete.py +++ b/Lib/idlelib/autocomplete.py @@ -9,6 +9,12 @@ import string import sys +# Modified keyword list is used in fetch_completions. +completion_kwds = [s for s in keyword.kwlist + if s not in {'True', 'False', 'None'}] # In builtins. +completion_kwds.extend(('match', 'case')) # Context keywords. +completion_kwds.sort() + # Two types of completions; defined here for autocomplete_w import below. ATTRS, FILES = 0, 1 from idlelib import autocomplete_w @@ -177,9 +183,7 @@ def fetch_completions(self, what, mode): namespace = {**__main__.__builtins__.__dict__, **__main__.__dict__} bigl = eval("dir()", namespace) - kwds = (s for s in keyword.kwlist - if s not in {'True', 'False', 'None'}) - bigl.extend(kwds) + bigl.extend(completion_kwds) bigl.sort() if "__all__" in bigl: smalll = sorted(eval("__all__", namespace)) diff --git a/Lib/idlelib/idle_test/test_autocomplete.py b/Lib/idlelib/idle_test/test_autocomplete.py index 642bb5db64dc3..a811363c18d04 100644 --- a/Lib/idlelib/idle_test/test_autocomplete.py +++ b/Lib/idlelib/idle_test/test_autocomplete.py @@ -218,6 +218,11 @@ def make_acw(): return self.dummy_acw() self.assertTrue(acp.open_completions(ac.TAB)) self.text.delete('1.0', 'end') + def test_completion_kwds(self): + self.assertIn('and', ac.completion_kwds) + self.assertIn('case', ac.completion_kwds) + self.assertNotIn('None', ac.completion_kwds) + def test_fetch_completions(self): # Test that fetch_completions returns 2 lists: # For attribute completion, a large list containing all variables, and diff --git a/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst b/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst new file mode 100644 index 0000000000000..3868f8d136a04 --- /dev/null +++ b/Misc/NEWS.d/next/IDLE/2021-10-16-17-20-32.bpo-45495.ST8RFt.rst @@ -0,0 +1 @@ +Add context keywords 'case' and 'match' to completions list. From webhook-mailer at python.org Sun Oct 17 07:59:30 2021 From: webhook-mailer at python.org (corona10) Date: Sun, 17 Oct 2021 11:59:30 -0000 Subject: [Python-checkins] Fix contributor person name in rst files (GH-29005) Message-ID: https://github.com/python/cpython/commit/a27f53bdd5b360fcfd94ca52e40f7c7e56597613 commit: a27f53bdd5b360fcfd94ca52e40f7c7e56597613 branch: main author: Benjamin Sz?ke committer: corona10 date: 2021-10-17T20:59:22+09:00 summary: Fix contributor person name in rst files (GH-29005) files: M Doc/whatsnew/3.11.rst M Misc/NEWS.d/3.11.0a1.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 9d7d3f19faae8..c589139b15488 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -261,12 +261,12 @@ time ``nanosleep()`` function, if available, which has a resolution of 1 nanosecond (10\ :sup:`-9` seconds), rather than using ``select()`` which has a resolution of 1 microsecond (10\ :sup:`-6` seconds). - (Contributed by Livius and Victor Stinner in :issue:`21302`.) + (Contributed by Benjamin Sz?ke and Victor Stinner in :issue:`21302`.) * On Windows, :func:`time.sleep` now uses a waitable timer which has a resolution of 100 nanoseconds (10\ :sup:`-7` seconds). Previously, it had a resolution of 1 millisecond (10\ :sup:`-3` seconds). - (Contributed by Livius and Victor Stinner in :issue:`21302`.) + (Contributed by Benjamin Sz?ke and Victor Stinner in :issue:`21302`.) unicodedata ----------- diff --git a/Misc/NEWS.d/3.11.0a1.rst b/Misc/NEWS.d/3.11.0a1.rst index a64a3e74ccb41..28b81b75266fc 100644 --- a/Misc/NEWS.d/3.11.0a1.rst +++ b/Misc/NEWS.d/3.11.0a1.rst @@ -1516,7 +1516,7 @@ available. ``nanosleep()`` allows to sleep with nanosecond precision. On Windows, :func:`time.sleep` now uses a waitable timer which has a resolution of 100 nanoseconds (10\ :sup:`-7` seconds). Previously, it had a -resolution of 1 millisecond (10\ :sup:`-3` seconds). Patch by Livius and +resolution of 1 millisecond (10\ :sup:`-3` seconds). Patch by Benjamin Sz?ke and Victor Stinner. .. From webhook-mailer at python.org Sun Oct 17 08:06:49 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 17 Oct 2021 12:06:49 -0000 Subject: [Python-checkins] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) Message-ID: https://github.com/python/cpython/commit/b3f0ceae919c1627094ff628c87184684a5cedd6 commit: b3f0ceae919c1627094ff628c87184684a5cedd6 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-17T15:06:40+03:00 summary: bpo-45229: Make test_http_cookiejar discoverable (GH-29004) files: M Lib/test/test_http_cookiejar.py diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index fdf15efde12ef..9450104d0b9a7 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -1920,14 +1920,5 @@ def test_session_cookies(self): self.assertNotEqual(counter["session_before"], 0) -def test_main(verbose=None): - test.support.run_unittest( - DateTimeTests, - HeaderTests, - CookieTests, - FileCookieJarTests, - LWPCookieTests, - ) - if __name__ == "__main__": - test_main(verbose=True) + unittest.main() From webhook-mailer at python.org Sun Oct 17 12:47:02 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Oct 2021 16:47:02 -0000 Subject: [Python-checkins] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) Message-ID: https://github.com/python/cpython/commit/1dbf9c86b25463b9bc695e434ac034a7e313ba01 commit: 1dbf9c86b25463b9bc695e434ac034a7e313ba01 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-17T09:46:55-07:00 summary: bpo-45229: Make test_http_cookiejar discoverable (GH-29004) (cherry picked from commit b3f0ceae919c1627094ff628c87184684a5cedd6) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_http_cookiejar.py diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index 2d7077af6da39..83945509d7148 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -1913,14 +1913,5 @@ def test_session_cookies(self): self.assertNotEqual(counter["session_before"], 0) -def test_main(verbose=None): - test.support.run_unittest( - DateTimeTests, - HeaderTests, - CookieTests, - FileCookieJarTests, - LWPCookieTests, - ) - if __name__ == "__main__": - test_main(verbose=True) + unittest.main() From webhook-mailer at python.org Sun Oct 17 12:47:07 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 17 Oct 2021 16:47:07 -0000 Subject: [Python-checkins] bpo-45229: Make test_http_cookiejar discoverable (GH-29004) Message-ID: https://github.com/python/cpython/commit/65c1db794e0484e18142658c691141387c1dc2e4 commit: 65c1db794e0484e18142658c691141387c1dc2e4 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-17T09:47:03-07:00 summary: bpo-45229: Make test_http_cookiejar discoverable (GH-29004) (cherry picked from commit b3f0ceae919c1627094ff628c87184684a5cedd6) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_http_cookiejar.py diff --git a/Lib/test/test_http_cookiejar.py b/Lib/test/test_http_cookiejar.py index fdf15efde12ef..9450104d0b9a7 100644 --- a/Lib/test/test_http_cookiejar.py +++ b/Lib/test/test_http_cookiejar.py @@ -1920,14 +1920,5 @@ def test_session_cookies(self): self.assertNotEqual(counter["session_before"], 0) -def test_main(verbose=None): - test.support.run_unittest( - DateTimeTests, - HeaderTests, - CookieTests, - FileCookieJarTests, - LWPCookieTests, - ) - if __name__ == "__main__": - test_main(verbose=True) + unittest.main() From webhook-mailer at python.org Sun Oct 17 19:20:54 2021 From: webhook-mailer at python.org (rhettinger) Date: Sun, 17 Oct 2021 23:20:54 -0000 Subject: [Python-checkins] Improve multiserver queue recipe (GH-29012) Message-ID: https://github.com/python/cpython/commit/54a4e1b53a18f0c7420ba03de9608194c4413fc2 commit: 54a4e1b53a18f0c7420ba03de9608194c4413fc2 branch: main author: Raymond Hettinger committer: rhettinger date: 2021-10-17T18:20:34-05:00 summary: Improve multiserver queue recipe (GH-29012) files: M Doc/library/random.rst diff --git a/Doc/library/random.rst b/Doc/library/random.rst index 0ac0fe72dc392..36f232dc319e4 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -503,7 +503,7 @@ between the effects of a drug versus a placebo:: Simulation of arrival times and service deliveries for a multiserver queue:: - from heapq import heappush, heappop + from heapq import heapify, heapreplace from random import expovariate, gauss from statistics import mean, quantiles @@ -515,14 +515,15 @@ Simulation of arrival times and service deliveries for a multiserver queue:: waits = [] arrival_time = 0.0 servers = [0.0] * num_servers # time when each server becomes available - for i in range(100_000): + heapify(servers) + for i in range(1_000_000): arrival_time += expovariate(1.0 / average_arrival_interval) - next_server_available = heappop(servers) + next_server_available = servers[0] wait = max(0.0, next_server_available - arrival_time) waits.append(wait) - service_duration = gauss(average_service_time, stdev_service_time) + service_duration = max(0.0, gauss(average_service_time, stdev_service_time)) service_completed = arrival_time + wait + service_duration - heappush(servers, service_completed) + heapreplace(servers, service_completed) print(f'Mean wait: {mean(waits):.1f} Max wait: {max(waits):.1f}') print('Quartiles:', [round(q, 1) for q in quantiles(waits)]) From webhook-mailer at python.org Sun Oct 17 19:43:18 2021 From: webhook-mailer at python.org (rhettinger) Date: Sun, 17 Oct 2021 23:43:18 -0000 Subject: [Python-checkins] Improve multiserver queue recipe (GH-29012) (GH-29013) Message-ID: https://github.com/python/cpython/commit/7082abf53db0ce599eb7347bd7efab0a647efb18 commit: 7082abf53db0ce599eb7347bd7efab0a647efb18 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: rhettinger date: 2021-10-17T18:43:10-05:00 summary: Improve multiserver queue recipe (GH-29012) (GH-29013) files: M Doc/library/random.rst diff --git a/Doc/library/random.rst b/Doc/library/random.rst index e924127d8b828..b9c33af59c8fa 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -512,7 +512,7 @@ between the effects of a drug versus a placebo:: Simulation of arrival times and service deliveries for a multiserver queue:: - from heapq import heappush, heappop + from heapq import heapify, heapreplace from random import expovariate, gauss from statistics import mean, quantiles @@ -524,14 +524,15 @@ Simulation of arrival times and service deliveries for a multiserver queue:: waits = [] arrival_time = 0.0 servers = [0.0] * num_servers # time when each server becomes available - for i in range(100_000): + heapify(servers) + for i in range(1_000_000): arrival_time += expovariate(1.0 / average_arrival_interval) - next_server_available = heappop(servers) + next_server_available = servers[0] wait = max(0.0, next_server_available - arrival_time) waits.append(wait) - service_duration = gauss(average_service_time, stdev_service_time) + service_duration = max(0.0, gauss(average_service_time, stdev_service_time)) service_completed = arrival_time + wait + service_duration - heappush(servers, service_completed) + heapreplace(servers, service_completed) print(f'Mean wait: {mean(waits):.1f} Max wait: {max(waits):.1f}') print('Quartiles:', [round(q, 1) for q in quantiles(waits)]) From webhook-mailer at python.org Mon Oct 18 01:25:00 2021 From: webhook-mailer at python.org (rhettinger) Date: Mon, 18 Oct 2021 05:25:00 -0000 Subject: [Python-checkins] Improve multiserver queue recipe (GH-29012) (GH-29014) Message-ID: https://github.com/python/cpython/commit/cb34c1ee1b1afe9165a26f1428362ea4bc6afe26 commit: cb34c1ee1b1afe9165a26f1428362ea4bc6afe26 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: rhettinger date: 2021-10-18T00:24:51-05:00 summary: Improve multiserver queue recipe (GH-29012) (GH-29014) files: M Doc/library/random.rst diff --git a/Doc/library/random.rst b/Doc/library/random.rst index f2d6749422a15..758d1292086e5 100644 --- a/Doc/library/random.rst +++ b/Doc/library/random.rst @@ -502,7 +502,7 @@ between the effects of a drug versus a placebo:: Simulation of arrival times and service deliveries for a multiserver queue:: - from heapq import heappush, heappop + from heapq import heapify, heapreplace from random import expovariate, gauss from statistics import mean, median, stdev @@ -514,14 +514,15 @@ Simulation of arrival times and service deliveries for a multiserver queue:: waits = [] arrival_time = 0.0 servers = [0.0] * num_servers # time when each server becomes available - for i in range(100_000): + heapify(servers) + for i in range(1_000_000): arrival_time += expovariate(1.0 / average_arrival_interval) - next_server_available = heappop(servers) + next_server_available = servers[0] wait = max(0.0, next_server_available - arrival_time) waits.append(wait) - service_duration = gauss(average_service_time, stdev_service_time) + service_duration = max(0.0, gauss(average_service_time, stdev_service_time)) service_completed = arrival_time + wait + service_duration - heappush(servers, service_completed) + heapreplace(servers, service_completed) print(f'Mean wait: {mean(waits):.1f}. Stdev wait: {stdev(waits):.1f}.') print(f'Median wait: {median(waits):.1f}. Max wait: {max(waits):.1f}.') From webhook-mailer at python.org Mon Oct 18 04:31:33 2021 From: webhook-mailer at python.org (corona10) Date: Mon, 18 Oct 2021 08:31:33 -0000 Subject: [Python-checkins] bpo-45434: Include stdlib.h for specialize stat (GH-29015) Message-ID: https://github.com/python/cpython/commit/fd03917786a9036ee87b7df604dfb260cc2420c9 commit: fd03917786a9036ee87b7df604dfb260cc2420c9 branch: main author: Dong-hee Na committer: corona10 date: 2021-10-18T17:31:18+09:00 summary: bpo-45434: Include stdlib.h for specialize stat (GH-29015) files: M Python/specialize.c diff --git a/Python/specialize.c b/Python/specialize.c index 529eabf6bc3ab..264637dc96856 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -1,4 +1,3 @@ - #include "Python.h" #include "pycore_code.h" #include "pycore_dict.h" @@ -8,6 +7,8 @@ #include "opcode.h" #include "structmember.h" // struct PyMemberDef, T_OFFSET_EX +#include // rand() + /* For guidance on adding or extending families of instructions see * ./adaptive.md */ From webhook-mailer at python.org Mon Oct 18 04:57:29 2021 From: webhook-mailer at python.org (markshannon) Date: Mon, 18 Oct 2021 08:57:29 -0000 Subject: [Python-checkins] bpo-45256: Avoid C calls for most Python to Python calls. (GH-28937) Message-ID: https://github.com/python/cpython/commit/70945d57e775b335eb58b734d82e68484063e835 commit: 70945d57e775b335eb58b734d82e68484063e835 branch: main author: Mark Shannon committer: markshannon date: 2021-10-18T09:57:24+01:00 summary: bpo-45256: Avoid C calls for most Python to Python calls. (GH-28937) * Avoid making C calls for most calls to Python functions. * Change initialize_locals(steal=true) and _PyTuple_FromArraySteal to consume the argument references regardless of whether they succeed or fail. files: M Lib/test/test_gdb.py M Objects/tupleobject.c M Python/ceval.c diff --git a/Lib/test/test_gdb.py b/Lib/test/test_gdb.py index fb0f1295574b9..2805eaf9f9567 100644 --- a/Lib/test/test_gdb.py +++ b/Lib/test/test_gdb.py @@ -724,24 +724,36 @@ def test_two_abs_args(self): ' 3 def foo(a, b, c):\n', bt) +SAMPLE_WITH_C_CALL = """ + +from _testcapi import pyobject_fastcall + +def foo(a, b, c): + bar(a, b, c) + +def bar(a, b, c): + pyobject_fastcall(baz, (a, b, c)) + +def baz(*args): + id(42) + +foo(1, 2, 3) + +""" + + class StackNavigationTests(DebuggerTests): @unittest.skipUnless(HAS_PYUP_PYDOWN, "test requires py-up/py-down commands") @unittest.skipIf(python_is_optimized(), "Python was compiled with optimizations") def test_pyup_command(self): 'Verify that the "py-up" command works' - bt = self.get_stack_trace(script=self.get_sample_script(), + bt = self.get_stack_trace(source=SAMPLE_WITH_C_CALL, cmds_after_breakpoint=['py-up', 'py-up']) self.assertMultilineMatches(bt, r'''^.* -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) - id\(42\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) - baz\(a, b, c\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\) - bar\(a=a, b=b, c=c\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in \(\) - foo\(1, 2, 3\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file , line 12, in baz \(args=\(1, 2, 3\)\) +#[0-9]+ $''') @unittest.skipUnless(HAS_PYUP_PYDOWN, "test requires py-up/py-down commands") @@ -765,22 +777,13 @@ def test_up_at_top(self): "Python was compiled with optimizations") def test_up_then_down(self): 'Verify "py-up" followed by "py-down"' - bt = self.get_stack_trace(script=self.get_sample_script(), + bt = self.get_stack_trace(source=SAMPLE_WITH_C_CALL, cmds_after_breakpoint=['py-up', 'py-up', 'py-down']) self.assertMultilineMatches(bt, r'''^.* -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) - id\(42\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) - baz\(a, b, c\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 4, in foo \(a=1, b=2, c=3\) - bar\(a=a, b=b, c=c\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 12, in \(\) - foo\(1, 2, 3\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 10, in baz \(args=\(1, 2, 3\)\) - id\(42\) -#[0-9]+ Frame 0x-?[0-9a-f]+, for file .*gdb_sample.py, line 7, in bar \(a=1, b=2, c=3\) - baz\(a, b, c\) +#[0-9]+ Frame 0x-?[0-9a-f]+, for file , line 12, in baz \(args=\(1, 2, 3\)\) +#[0-9]+ +#[0-9]+ Frame 0x-?[0-9a-f]+, for file , line 12, in baz \(args=\(1, 2, 3\)\) $''') class PyBtTests(DebuggerTests): @@ -970,13 +973,12 @@ def __init__(self): self.assertRegex(gdb_output, r"ob_item; diff --git a/Python/ceval.c b/Python/ceval.c index 898687224e696..60fdf01627eba 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -50,9 +50,9 @@ _Py_IDENTIFIER(__name__); /* Forward declarations */ -Py_LOCAL_INLINE(PyObject *) call_function( - PyThreadState *tstate, PyObject ***pp_stack, - Py_ssize_t oparg, PyObject *kwnames, int use_tracing); +static PyObject *trace_call_function( + PyThreadState *tstate, PyObject *callable, PyObject **stack, + Py_ssize_t oparg, PyObject *kwnames); static PyObject * do_call_core( PyThreadState *tstate, PyObject *func, PyObject *callargs, PyObject *kwdict, int use_tracing); @@ -1702,6 +1702,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr switch (opcode) { #endif + /* Variables used for making calls */ + PyObject *kwnames; + int nargs; + int postcall_shrink; + /* BEWARE! It is essential that any operation that fails must goto error and that all operation that succeed call DISPATCH() ! */ @@ -4599,10 +4604,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(CALL_METHOD) { /* Designed to work in tamdem with LOAD_METHOD. */ - PyObject **sp, *res; - int meth_found; - - sp = stack_pointer; /* `meth` is NULL when LOAD_METHOD thinks that it's not a method call. @@ -4628,114 +4629,84 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr We'll be passing `oparg + 1` to call_function, to make it accept the `self` as a first argument. */ - meth_found = (PEEK(oparg + 2) != NULL); - res = call_function(tstate, &sp, oparg + meth_found, NULL, cframe.use_tracing); - stack_pointer = sp; - - STACK_SHRINK(1 - meth_found); - PUSH(res); - if (res == NULL) { - goto error; - } - CHECK_EVAL_BREAKER(); - DISPATCH(); + int is_method = (PEEK(oparg + 2) != NULL); + oparg += is_method; + nargs = oparg; + kwnames = NULL; + postcall_shrink = 2-is_method; + goto call_function; } TARGET(CALL_METHOD_KW) { /* Designed to work in tandem with LOAD_METHOD. Same as CALL_METHOD but pops TOS to get a tuple of keyword names. */ - PyObject **sp, *res; - PyObject *names = NULL; - int meth_found; - - names = POP(); - - sp = stack_pointer; - meth_found = (PEEK(oparg + 2) != NULL); - res = call_function(tstate, &sp, oparg + meth_found, names, cframe.use_tracing); - stack_pointer = sp; + kwnames = POP(); + int is_method = (PEEK(oparg + 2) != NULL); + oparg += is_method; + nargs = oparg - (int)PyTuple_GET_SIZE(kwnames); + postcall_shrink = 2-is_method; + goto call_function; + } - STACK_SHRINK(1 - meth_found); - PUSH(res); - Py_DECREF(names); - if (res == NULL) { - goto error; - } - CHECK_EVAL_BREAKER(); - DISPATCH(); + TARGET(CALL_FUNCTION_KW) { + kwnames = POP(); + nargs = oparg - (int)PyTuple_GET_SIZE(kwnames); + postcall_shrink = 1; + goto call_function; } TARGET(CALL_FUNCTION) { PREDICTED(CALL_FUNCTION); - PyObject *res; - + PyObject *function; + nargs = oparg; + kwnames = NULL; + postcall_shrink = 1; + call_function: // Check if the call can be inlined or not - PyObject *function = PEEK(oparg + 1); + function = PEEK(oparg + 1); if (Py_TYPE(function) == &PyFunction_Type && tstate->interp->eval_frame == NULL) { int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(function))->co_flags; - PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); int is_generator = code_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR); if (!is_generator) { + PyObject *locals = code_flags & CO_OPTIMIZED ? NULL : PyFunction_GET_GLOBALS(function); + STACK_SHRINK(oparg); InterpreterFrame *new_frame = _PyEvalFramePushAndInit( tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, - stack_pointer-oparg, - oparg, NULL, 1); - if (new_frame == NULL) { - // When we exit here, we own all variables in the stack - // (the frame creation has not stolen any variable) so - // we need to clean the whole stack (done in the - // "error" label). - goto error; - } - - STACK_SHRINK(oparg + 1); + stack_pointer, + nargs, kwnames, 1); + STACK_SHRINK(postcall_shrink); // The frame has stolen all the arguments from the stack, // so there is no need to clean them up. + Py_XDECREF(kwnames); Py_DECREF(function); + if (new_frame == NULL) { + goto error; + } _PyFrame_SetStackPointer(frame, stack_pointer); new_frame->depth = frame->depth + 1; tstate->frame = frame = new_frame; goto start_frame; } - else { - /* Callable is a generator or coroutine function: create - * coroutine or generator. */ - res = make_coro(tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), - locals, stack_pointer-oparg, oparg, NULL); - STACK_SHRINK(oparg + 1); - for (int i = 0; i < oparg + 1; i++) { - Py_DECREF(stack_pointer[i]); - } - } + } + /* Callable is not a normal Python function */ + PyObject *res; + if (cframe.use_tracing) { + res = trace_call_function(tstate, function, stack_pointer-oparg, nargs, kwnames); } else { - /* Callable is not a Python function */ - PyObject **sp = stack_pointer; - res = call_function(tstate, &sp, oparg, NULL, cframe.use_tracing); - stack_pointer = sp; + res = PyObject_Vectorcall(function, stack_pointer-oparg, + nargs | PY_VECTORCALL_ARGUMENTS_OFFSET, kwnames); } - - PUSH(res); - if (res == NULL) { - goto error; + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + Py_DECREF(function); + Py_XDECREF(kwnames); + /* Clear the stack */ + STACK_SHRINK(oparg); + for (int i = 0; i < oparg; i++) { + Py_DECREF(stack_pointer[i]); } - CHECK_EVAL_BREAKER(); - DISPATCH(); - } - - TARGET(CALL_FUNCTION_KW) { - PyObject **sp, *res, *names; - - names = POP(); - assert(PyTuple_Check(names)); - assert(PyTuple_GET_SIZE(names) <= oparg); - /* We assume without checking that names contains only strings */ - sp = stack_pointer; - res = call_function(tstate, &sp, oparg, names, cframe.use_tracing); - stack_pointer = sp; + STACK_SHRINK(postcall_shrink); PUSH(res); - Py_DECREF(names); - if (res == NULL) { goto error; } @@ -5482,7 +5453,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (co->co_flags & CO_VARKEYWORDS) { kwdict = PyDict_New(); if (kwdict == NULL) { - goto fail; + goto fail_pre_positional; } i = total_args; if (co->co_flags & CO_VARARGS) { @@ -5521,11 +5492,19 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, u = _PyTuple_FromArray(args + n, argcount - n); } if (u == NULL) { - goto fail; + goto fail_post_positional; } assert(localsplus[total_args] == NULL); localsplus[total_args] = u; } + else if (argcount > n) { + /* Too many postional args. Error is reported later */ + if (steal_args) { + for (j = n; j < argcount; j++) { + Py_DECREF(args[j]); + } + } + } /* Handle keyword arguments */ if (kwnames != NULL) { @@ -5540,7 +5519,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, _PyErr_Format(tstate, PyExc_TypeError, "%U() keywords must be strings", con->fc_qualname); - goto fail; + goto kw_fail; } /* Speed hack: do raw pointer compares. As names are @@ -5561,7 +5540,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, goto kw_found; } else if (cmp < 0) { - goto fail; + goto kw_fail; } } @@ -5573,29 +5552,38 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, kwcount, kwnames, con->fc_qualname)) { - goto fail; + goto kw_fail; } _PyErr_Format(tstate, PyExc_TypeError, "%U() got an unexpected keyword argument '%S'", con->fc_qualname, keyword); - goto fail; + goto kw_fail; } if (PyDict_SetItem(kwdict, keyword, value) == -1) { - goto fail; + goto kw_fail; } if (steal_args) { Py_DECREF(value); } continue; + kw_fail: + if (steal_args) { + for (;i < kwcount; i++) { + PyObject *value = args[i+argcount]; + Py_DECREF(value); + } + } + goto fail_noclean; + kw_found: if (localsplus[j] != NULL) { _PyErr_Format(tstate, PyExc_TypeError, "%U() got multiple values for argument '%S'", con->fc_qualname, keyword); - goto fail; + goto kw_fail; } if (!steal_args) { Py_INCREF(value); @@ -5608,7 +5596,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if ((argcount > co->co_argcount) && !(co->co_flags & CO_VARARGS)) { too_many_positional(tstate, co, argcount, con->fc_defaults, localsplus, con->fc_qualname); - goto fail; + goto fail_noclean; } /* Add missing positional arguments (copy default values from defs) */ @@ -5624,7 +5612,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (missing) { missing_arguments(tstate, co, missing, defcount, localsplus, con->fc_qualname); - goto fail; + goto fail_noclean; } if (n > m) i = n - m; @@ -5657,7 +5645,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, continue; } else if (_PyErr_Occurred(tstate)) { - goto fail; + goto fail_noclean; } } missing++; @@ -5665,7 +5653,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (missing) { missing_arguments(tstate, co, missing, -1, localsplus, con->fc_qualname); - goto fail; + goto fail_noclean; } } @@ -5678,33 +5666,23 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, return 0; -fail: /* Jump here from prelude on failure */ +fail_pre_positional: if (steal_args) { - // If we failed to initialize locals, make sure the caller still own all the - // arguments that were on the stack. We need to increment the reference count - // of everything we copied (everything in localsplus) that came from the stack - // (everything that is present in the "args" array). - Py_ssize_t kwcount = kwnames != NULL ? PyTuple_GET_SIZE(kwnames) : 0; - for (Py_ssize_t k=0; k < total_args; k++) { - PyObject* arg = localsplus[k]; - for (Py_ssize_t j=0; j < argcount + kwcount; j++) { - if (args[j] == arg) { - Py_XINCREF(arg); - break; - } - } + for (j = 0; j < argcount; j++) { + Py_DECREF(args[j]); } - // Restore all the **kwargs we placed into the kwargs dictionary - if (kwdict) { - PyObject *key, *value; - Py_ssize_t pos = 0; - while (PyDict_Next(kwdict, &pos, &key, &value)) { - Py_INCREF(value); - } + } + /* fall through */ +fail_post_positional: + if (steal_args) { + Py_ssize_t kwcount = kwnames != NULL ? PyTuple_GET_SIZE(kwnames) : 0; + for (j = argcount; j < argcount+kwcount; j++) { + Py_DECREF(args[j]); } } + /* fall through */ +fail_noclean: return -1; - } static InterpreterFrame * @@ -6593,40 +6571,6 @@ trace_call_function(PyThreadState *tstate, return PyObject_Vectorcall(func, args, nargs | PY_VECTORCALL_ARGUMENTS_OFFSET, kwnames); } -/* Issue #29227: Inline call_function() into _PyEval_EvalFrameDefault() - to reduce the stack consumption. */ -Py_LOCAL_INLINE(PyObject *) _Py_HOT_FUNCTION -call_function(PyThreadState *tstate, - PyObject ***pp_stack, - Py_ssize_t oparg, - PyObject *kwnames, - int use_tracing) -{ - PyObject **pfunc = (*pp_stack) - oparg - 1; - PyObject *func = *pfunc; - PyObject *x, *w; - Py_ssize_t nkwargs = (kwnames == NULL) ? 0 : PyTuple_GET_SIZE(kwnames); - Py_ssize_t nargs = oparg - nkwargs; - PyObject **stack = (*pp_stack) - nargs - nkwargs; - - if (use_tracing) { - x = trace_call_function(tstate, func, stack, nargs, kwnames); - } - else { - x = PyObject_Vectorcall(func, stack, nargs | PY_VECTORCALL_ARGUMENTS_OFFSET, kwnames); - } - - assert((x != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); - - /* Clear the stack of the function object. */ - while ((*pp_stack) > pfunc) { - w = EXT_POP(*pp_stack); - Py_DECREF(w); - } - - return x; -} - static PyObject * do_call_core(PyThreadState *tstate, PyObject *func, From webhook-mailer at python.org Mon Oct 18 12:40:52 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 18 Oct 2021 16:40:52 -0000 Subject: [Python-checkins] bpo-43760: Rename _PyThreadState_DisableTracing() (GH-29032) Message-ID: https://github.com/python/cpython/commit/034f607906de6e375cc9a98cc3b09f6d56f8be10 commit: 034f607906de6e375cc9a98cc3b09f6d56f8be10 branch: main author: Victor Stinner committer: vstinner date: 2021-10-18T18:40:43+02:00 summary: bpo-43760: Rename _PyThreadState_DisableTracing() (GH-29032) * Rename _PyThreadState_DisableTracing() to _PyThreadState_PauseTracing() * Rename _PyThreadState_ResetTracing() to _PyThreadState_ResumeTracing() files: M Include/internal/pycore_pystate.h M Python/ceval.c M Python/pystate.c diff --git a/Include/internal/pycore_pystate.h b/Include/internal/pycore_pystate.h index d63239736caba..9a570b08bc583 100644 --- a/Include/internal/pycore_pystate.h +++ b/Include/internal/pycore_pystate.h @@ -134,13 +134,13 @@ PyAPI_FUNC(void) _PyThreadState_DeleteExcept( PyThreadState *tstate); static inline void -_PyThreadState_DisableTracing(PyThreadState *tstate) +_PyThreadState_PauseTracing(PyThreadState *tstate) { tstate->cframe->use_tracing = 0; } static inline void -_PyThreadState_ResetTracing(PyThreadState *tstate) +_PyThreadState_ResumeTracing(PyThreadState *tstate) { int use_tracing = (tstate->c_tracefunc != NULL || tstate->c_profilefunc != NULL); diff --git a/Python/ceval.c b/Python/ceval.c index 60fdf01627eba..05bdcc5f7fab9 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -6165,7 +6165,7 @@ call_trace(Py_tracefunc func, PyObject *obj, if (tstate->tracing) return 0; tstate->tracing++; - _PyThreadState_DisableTracing(tstate); + _PyThreadState_PauseTracing(tstate); PyFrameObject *f = _PyFrame_GetFrameObject(frame); if (f == NULL) { return -1; @@ -6179,7 +6179,7 @@ call_trace(Py_tracefunc func, PyObject *obj, } result = func(obj, f, what, arg); f->f_lineno = 0; - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); tstate->tracing--; return result; } @@ -6193,7 +6193,7 @@ _PyEval_CallTracing(PyObject *func, PyObject *args) PyObject *result; tstate->tracing = 0; - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); result = PyObject_Call(func, args, NULL); tstate->tracing = save_tracing; tstate->cframe->use_tracing = save_use_tracing; @@ -6250,7 +6250,7 @@ _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_profilefunc = NULL; tstate->c_profileobj = NULL; /* Must make sure that tracing is not ignored if 'profileobj' is freed */ - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); Py_XDECREF(profileobj); Py_XINCREF(arg); @@ -6258,7 +6258,7 @@ _PyEval_SetProfile(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_profilefunc = func; /* Flag that tracing or profiling is turned on */ - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); return 0; } @@ -6291,7 +6291,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_tracefunc = NULL; tstate->c_traceobj = NULL; /* Must make sure that profiling is not ignored if 'traceobj' is freed */ - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); Py_XDECREF(traceobj); Py_XINCREF(arg); @@ -6299,7 +6299,7 @@ _PyEval_SetTrace(PyThreadState *tstate, Py_tracefunc func, PyObject *arg) tstate->c_tracefunc = func; /* Flag that tracing or profiling is turned on */ - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); return 0; } diff --git a/Python/pystate.c b/Python/pystate.c index abd711ee2e796..7804e17a064e1 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -1205,14 +1205,14 @@ void PyThreadState_EnterTracing(PyThreadState *tstate) { tstate->tracing++; - _PyThreadState_DisableTracing(tstate); + _PyThreadState_PauseTracing(tstate); } void PyThreadState_LeaveTracing(PyThreadState *tstate) { tstate->tracing--; - _PyThreadState_ResetTracing(tstate); + _PyThreadState_ResumeTracing(tstate); } From webhook-mailer at python.org Mon Oct 18 14:26:31 2021 From: webhook-mailer at python.org (ned-deily) Date: Mon, 18 Oct 2021 18:26:31 -0000 Subject: [Python-checkins] bpo-45221: Fix handling of LDFLAGS and CPPFLAGS options in setup.py (GH-29031) Message-ID: https://github.com/python/cpython/commit/6a533a423869e28d9086cf4d79029f59e9eec916 commit: 6a533a423869e28d9086cf4d79029f59e9eec916 branch: main author: andrei kulakov committer: ned-deily date: 2021-10-18T14:26:23-04:00 summary: bpo-45221: Fix handling of LDFLAGS and CPPFLAGS options in setup.py (GH-29031) files: A Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst M setup.py diff --git a/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst b/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst new file mode 100644 index 0000000000000..cb981d96f3047 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst @@ -0,0 +1,3 @@ +Fixed regression in handling of ``LDFLAGS`` and ``CPPFLAGS`` options +where :meth:`argparse.parse_known_args` could interpret an option as +one of the built-in command line argument, for example ``-h`` for help. diff --git a/setup.py b/setup.py index 24365ef3d927b..5428cbde1cc9b 100644 --- a/setup.py +++ b/setup.py @@ -801,6 +801,18 @@ def add_ldflags_cppflags(self): if env_val: parser = argparse.ArgumentParser() parser.add_argument(arg_name, dest="dirs", action="append") + + # To prevent argparse from raising an exception about any + # options in env_val that it mistakes for known option, we + # strip out all double dashes and any dashes followed by a + # character that is not for the option we are dealing with. + # + # Please note that order of the regex is important! We must + # strip out double-dashes first so that we don't end up with + # substituting "--Long" to "-Long" and thus lead to "ong" being + # used for a library directory. + env_val = re.sub(r'(^|\s+)-(-|(?!%s))' % arg_name[1], + ' ', env_val) options, _ = parser.parse_known_args(env_val.split()) if options.dirs: for directory in reversed(options.dirs): From webhook-mailer at python.org Mon Oct 18 14:49:37 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 18 Oct 2021 18:49:37 -0000 Subject: [Python-checkins] [3.10] bpo-45221: Fix handling of LDFLAGS and CPPFLAGS options in setup.py (GH-29031) (GH-29037) Message-ID: https://github.com/python/cpython/commit/b1949e0b58714724a3105cad3ad1b61384688da7 commit: b1949e0b58714724a3105cad3ad1b61384688da7 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-18T11:49:28-07:00 summary: [3.10] bpo-45221: Fix handling of LDFLAGS and CPPFLAGS options in setup.py (GH-29031) (GH-29037) (cherry picked from commit 6a533a423869e28d9086cf4d79029f59e9eec916) Co-authored-by: andrei kulakov Automerge-Triggered-By: GH:ned-deily files: A Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst M setup.py diff --git a/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst b/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst new file mode 100644 index 0000000000000..cb981d96f3047 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-18-10-25-56.bpo-45221.rnulhf.rst @@ -0,0 +1,3 @@ +Fixed regression in handling of ``LDFLAGS`` and ``CPPFLAGS`` options +where :meth:`argparse.parse_known_args` could interpret an option as +one of the built-in command line argument, for example ``-h`` for help. diff --git a/setup.py b/setup.py index a6fcc12b84d0c..74d5cefc60ee3 100644 --- a/setup.py +++ b/setup.py @@ -801,6 +801,18 @@ def add_ldflags_cppflags(self): if env_val: parser = argparse.ArgumentParser() parser.add_argument(arg_name, dest="dirs", action="append") + + # To prevent argparse from raising an exception about any + # options in env_val that it mistakes for known option, we + # strip out all double dashes and any dashes followed by a + # character that is not for the option we are dealing with. + # + # Please note that order of the regex is important! We must + # strip out double-dashes first so that we don't end up with + # substituting "--Long" to "-Long" and thus lead to "ong" being + # used for a library directory. + env_val = re.sub(r'(^|\s+)-(-|(?!%s))' % arg_name[1], + ' ', env_val) options, _ = parser.parse_known_args(env_val.split()) if options.dirs: for directory in reversed(options.dirs): From webhook-mailer at python.org Mon Oct 18 18:58:24 2021 From: webhook-mailer at python.org (jaraco) Date: Mon, 18 Oct 2021 22:58:24 -0000 Subject: [Python-checkins] bpo-45516: add protocol description to the Traversable documentation (#29039) Message-ID: https://github.com/python/cpython/commit/4d03de3329ed8daa9c1107b1aedbb0fa280bddb6 commit: 4d03de3329ed8daa9c1107b1aedbb0fa280bddb6 branch: main author: Filipe La?ns committer: jaraco date: 2021-10-18T18:58:13-04:00 summary: bpo-45516: add protocol description to the Traversable documentation (#29039) * bpo-45516: add protocol description to the Traversable documentation Signed-off-by: Filipe La?ns * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs * Update Lib/importlib/abc.py * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs files: A Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index b5ee7a6b9659a..3576941efa46d 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -815,6 +815,46 @@ ABC hierarchy:: .. versionadded:: 3.9 + .. abstractmethod:: name() + + The base name of this object without any parent references. + + .. abstractmethod:: iterdir() + + Yield Traversable objects in self. + + .. abstractmethod:: is_dir() + + Return True if self is a directory. + + .. abstractmethod:: is_file() + + Return True if self is a file. + + .. abstractmethod:: joinpath(child) + + Return Traversable child in self. + + .. abstractmethod:: __truediv__(child) + + Return Traversable child in self. + + .. abstractmethod:: open(mode='r', *args, **kwargs) + + *mode* may be 'r' or 'rb' to open as text or binary. Return a handle + suitable for reading (same as :attr:`pathlib.Path.open`). + + When opening as text, accepts encoding parameters such as those + accepted by :attr:`io.TextIOWrapper`. + + .. method:: read_bytes() + + Read contents of self as bytes. + + .. method:: read_text(encoding=None) + + Read contents of self as text. + .. class:: TraversableResources diff --git a/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst new file mode 100644 index 0000000000000..98f5d3432db05 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst @@ -0,0 +1,2 @@ +Add protocol description to the :class:`importlib.abc.Traversable` +documentation. From webhook-mailer at python.org Mon Oct 18 19:32:06 2021 From: webhook-mailer at python.org (vstinner) Date: Mon, 18 Oct 2021 23:32:06 -0000 Subject: [Python-checkins] bpo-35134: Split warnings.h and weakrefobject.h (GH-29042) Message-ID: https://github.com/python/cpython/commit/aad88d33d9db0a93e480f0234292b948890dfc2a commit: aad88d33d9db0a93e480f0234292b948890dfc2a branch: main author: Victor Stinner committer: vstinner date: 2021-10-19T01:31:57+02:00 summary: bpo-35134: Split warnings.h and weakrefobject.h (GH-29042) Split header files to move the non-limited API to Include/cpython/: * Include/warnings.h => Include/cpython/warnings.h * Include/weakrefobject.h => Include/cpython/weakrefobject.h Exclude PyWeakref_GET_OBJECT() from the limited C API. It never worked since the PyWeakReference structure is opaque in the limited C API. Move _PyWarnings_Init() and _PyErr_WarnUnawaitedCoroutine() to the internal C API. files: A Include/cpython/warnings.h A Include/cpython/weakrefobject.h A Misc/NEWS.d/next/C API/2021-10-19-00-20-40.bpo-35134.Z0Zk_m.rst M Doc/whatsnew/3.11.rst M Include/internal/pycore_warnings.h M Include/warnings.h M Include/weakrefobject.h M Makefile.pre.in M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index c589139b15488..0776265779849 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -644,3 +644,8 @@ Removed Remove also the ``Py_MARSHAL_VERSION`` macro from the limited C API. (Contributed by Victor Stinner in :issue:`45474`.) + +* Exclude :c:func:`PyWeakref_GET_OBJECT` from the limited C API. It never + worked since the :c:type:`PyWeakReference` structure is opaque in the + limited C API. + (Contributed by Victor Stinner in :issue:`35134`.) diff --git a/Include/cpython/warnings.h b/Include/cpython/warnings.h new file mode 100644 index 0000000000000..2ef8e3ce9435f --- /dev/null +++ b/Include/cpython/warnings.h @@ -0,0 +1,20 @@ +#ifndef Py_CPYTHON_WARNINGS_H +# error "this header file must not be included directly" +#endif + +PyAPI_FUNC(int) PyErr_WarnExplicitObject( + PyObject *category, + PyObject *message, + PyObject *filename, + int lineno, + PyObject *module, + PyObject *registry); + +PyAPI_FUNC(int) PyErr_WarnExplicitFormat( + PyObject *category, + const char *filename, int lineno, + const char *module, PyObject *registry, + const char *format, ...); + +// DEPRECATED: Use PyErr_WarnEx() instead. +#define PyErr_Warn(category, msg) PyErr_WarnEx(category, msg, 1) diff --git a/Include/cpython/weakrefobject.h b/Include/cpython/weakrefobject.h new file mode 100644 index 0000000000000..9efcc412df9be --- /dev/null +++ b/Include/cpython/weakrefobject.h @@ -0,0 +1,47 @@ +#ifndef Py_CPYTHON_WEAKREFOBJECT_H +# error "this header file must not be included directly" +#endif + +/* PyWeakReference is the base struct for the Python ReferenceType, ProxyType, + * and CallableProxyType. + */ +struct _PyWeakReference { + PyObject_HEAD + + /* The object to which this is a weak reference, or Py_None if none. + * Note that this is a stealth reference: wr_object's refcount is + * not incremented to reflect this pointer. + */ + PyObject *wr_object; + + /* A callable to invoke when wr_object dies, or NULL if none. */ + PyObject *wr_callback; + + /* A cache for wr_object's hash code. As usual for hashes, this is -1 + * if the hash code isn't known yet. + */ + Py_hash_t hash; + + /* If wr_object is weakly referenced, wr_object has a doubly-linked NULL- + * terminated list of weak references to it. These are the list pointers. + * If wr_object goes away, wr_object is set to Py_None, and these pointers + * have no meaning then. + */ + PyWeakReference *wr_prev; + PyWeakReference *wr_next; +}; + +PyAPI_FUNC(Py_ssize_t) _PyWeakref_GetWeakrefCount(PyWeakReference *head); + +PyAPI_FUNC(void) _PyWeakref_ClearRef(PyWeakReference *self); + +/* Explanation for the Py_REFCNT() check: when a weakref's target is part + of a long chain of deallocations which triggers the trashcan mechanism, + clearing the weakrefs can be delayed long after the target's refcount + has dropped to zero. In the meantime, code accessing the weakref will + be able to "see" the target object even though it is supposed to be + unreachable. See issue #16602. */ +#define PyWeakref_GET_OBJECT(ref) \ + (Py_REFCNT(((PyWeakReference *)(ref))->wr_object) > 0 \ + ? ((PyWeakReference *)(ref))->wr_object \ + : Py_None) diff --git a/Include/internal/pycore_warnings.h b/Include/internal/pycore_warnings.h index f728ec3077b3c..efb4f1cd7eac8 100644 --- a/Include/internal/pycore_warnings.h +++ b/Include/internal/pycore_warnings.h @@ -19,6 +19,10 @@ struct _warnings_runtime_state { extern int _PyWarnings_InitState(PyInterpreterState *interp); +PyAPI_FUNC(PyObject*) _PyWarnings_Init(void); + +extern void _PyErr_WarnUnawaitedCoroutine(PyObject *coro); + #ifdef __cplusplus } #endif diff --git a/Include/warnings.h b/Include/warnings.h index a675bb5dfcb9f..18ac1543a3ca9 100644 --- a/Include/warnings.h +++ b/Include/warnings.h @@ -4,14 +4,11 @@ extern "C" { #endif -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject*) _PyWarnings_Init(void); -#endif - PyAPI_FUNC(int) PyErr_WarnEx( PyObject *category, const char *message, /* UTF-8 encoded string */ Py_ssize_t stack_level); + PyAPI_FUNC(int) PyErr_WarnFormat( PyObject *category, Py_ssize_t stack_level, @@ -26,15 +23,7 @@ PyAPI_FUNC(int) PyErr_ResourceWarning( const char *format, /* ASCII-encoded string */ ...); #endif -#ifndef Py_LIMITED_API -PyAPI_FUNC(int) PyErr_WarnExplicitObject( - PyObject *category, - PyObject *message, - PyObject *filename, - int lineno, - PyObject *module, - PyObject *registry); -#endif + PyAPI_FUNC(int) PyErr_WarnExplicit( PyObject *category, const char *message, /* UTF-8 encoded string */ @@ -44,20 +33,9 @@ PyAPI_FUNC(int) PyErr_WarnExplicit( PyObject *registry); #ifndef Py_LIMITED_API -PyAPI_FUNC(int) -PyErr_WarnExplicitFormat(PyObject *category, - const char *filename, int lineno, - const char *module, PyObject *registry, - const char *format, ...); -#endif - -/* DEPRECATED: Use PyErr_WarnEx() instead. */ -#ifndef Py_LIMITED_API -#define PyErr_Warn(category, msg) PyErr_WarnEx(category, msg, 1) -#endif - -#ifndef Py_LIMITED_API -void _PyErr_WarnUnawaitedCoroutine(PyObject *coro); +# define Py_CPYTHON_WARNINGS_H +# include "cpython/warnings.h" +# undef Py_CPYTHON_WARNINGS_H #endif #ifdef __cplusplus diff --git a/Include/weakrefobject.h b/Include/weakrefobject.h index ac4b4821c8a14..f071e9c759a64 100644 --- a/Include/weakrefobject.h +++ b/Include/weakrefobject.h @@ -6,40 +6,8 @@ extern "C" { #endif - typedef struct _PyWeakReference PyWeakReference; -/* PyWeakReference is the base struct for the Python ReferenceType, ProxyType, - * and CallableProxyType. - */ -#ifndef Py_LIMITED_API -struct _PyWeakReference { - PyObject_HEAD - - /* The object to which this is a weak reference, or Py_None if none. - * Note that this is a stealth reference: wr_object's refcount is - * not incremented to reflect this pointer. - */ - PyObject *wr_object; - - /* A callable to invoke when wr_object dies, or NULL if none. */ - PyObject *wr_callback; - - /* A cache for wr_object's hash code. As usual for hashes, this is -1 - * if the hash code isn't known yet. - */ - Py_hash_t hash; - - /* If wr_object is weakly referenced, wr_object has a doubly-linked NULL- - * terminated list of weak references to it. These are the list pointers. - * If wr_object goes away, wr_object is set to Py_None, and these pointers - * have no meaning then. - */ - PyWeakReference *wr_prev; - PyWeakReference *wr_next; -}; -#endif - PyAPI_DATA(PyTypeObject) _PyWeakref_RefType; PyAPI_DATA(PyTypeObject) _PyWeakref_ProxyType; PyAPI_DATA(PyTypeObject) _PyWeakref_CallableProxyType; @@ -56,30 +24,18 @@ PyAPI_DATA(PyTypeObject) _PyWeakref_CallableProxyType; PyAPI_FUNC(PyObject *) PyWeakref_NewRef(PyObject *ob, - PyObject *callback); + PyObject *callback); PyAPI_FUNC(PyObject *) PyWeakref_NewProxy(PyObject *ob, - PyObject *callback); + PyObject *callback); PyAPI_FUNC(PyObject *) PyWeakref_GetObject(PyObject *ref); -#ifndef Py_LIMITED_API -PyAPI_FUNC(Py_ssize_t) _PyWeakref_GetWeakrefCount(PyWeakReference *head); -PyAPI_FUNC(void) _PyWeakref_ClearRef(PyWeakReference *self); +#ifndef Py_LIMITED_API +# define Py_CPYTHON_WEAKREFOBJECT_H +# include "cpython/weakrefobject.h" +# undef Py_CPYTHON_WEAKREFOBJECT_H #endif -/* Explanation for the Py_REFCNT() check: when a weakref's target is part - of a long chain of deallocations which triggers the trashcan mechanism, - clearing the weakrefs can be delayed long after the target's refcount - has dropped to zero. In the meantime, code accessing the weakref will - be able to "see" the target object even though it is supposed to be - unreachable. See issue #16602. */ - -#define PyWeakref_GET_OBJECT(ref) \ - (Py_REFCNT(((PyWeakReference *)(ref))->wr_object) > 0 \ - ? ((PyWeakReference *)(ref))->wr_object \ - : Py_None) - - #ifdef __cplusplus } #endif diff --git a/Makefile.pre.in b/Makefile.pre.in index 9de51711ac4c5..60dc22489dd66 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1228,6 +1228,8 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/traceback.h \ $(srcdir)/Include/cpython/tupleobject.h \ $(srcdir)/Include/cpython/unicodeobject.h \ + $(srcdir)/Include/cpython/warnings.h \ + $(srcdir)/Include/cpython/weakrefobject.h \ \ $(srcdir)/Include/internal/pycore_abstract.h \ $(srcdir)/Include/internal/pycore_accu.h \ diff --git a/Misc/NEWS.d/next/C API/2021-10-19-00-20-40.bpo-35134.Z0Zk_m.rst b/Misc/NEWS.d/next/C API/2021-10-19-00-20-40.bpo-35134.Z0Zk_m.rst new file mode 100644 index 0000000000000..57c4fa53bee66 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-19-00-20-40.bpo-35134.Z0Zk_m.rst @@ -0,0 +1,3 @@ +Exclude :c:func:`PyWeakref_GET_OBJECT` from the limited C API. It never +worked since the :c:type:`PyWeakReference` structure is opaque in the +limited C API. diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 015d783c89a83..a26ba9c232e41 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -160,6 +160,8 @@ + + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 94528b9261fa6..335bfb31bed54 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -393,6 +393,12 @@ Include\cpython + + Include\cpython + + + Include\cpython + Include\cpython From webhook-mailer at python.org Mon Oct 18 20:05:04 2021 From: webhook-mailer at python.org (vstinner) Date: Tue, 19 Oct 2021 00:05:04 -0000 Subject: [Python-checkins] bpo-35134: Add Include/cpython/longobject.h (GH-29044) Message-ID: https://github.com/python/cpython/commit/5f09bb021a2862ba89c3ecb53e7e6e95a9e07e1d commit: 5f09bb021a2862ba89c3ecb53e7e6e95a9e07e1d branch: main author: Victor Stinner committer: vstinner date: 2021-10-19T02:04:52+02:00 summary: bpo-35134: Add Include/cpython/longobject.h (GH-29044) Move Include/longobject.h non-limited API to a new Include/cpython/longobject.h header file. Move the following definitions to the internal C API: * _PyLong_DigitValue * _PyLong_FormatAdvancedWriter() * _PyLong_FormatWriter() files: A Include/cpython/longobject.h M Include/internal/pycore_long.h M Include/longobject.h M Makefile.pre.in M Modules/binascii.c M Objects/bytesobject.c M Objects/unicodeobject.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M Python/formatter_unicode.c M Python/mystrtoul.c diff --git a/Include/cpython/longobject.h b/Include/cpython/longobject.h new file mode 100644 index 0000000000000..1a73799d658fe --- /dev/null +++ b/Include/cpython/longobject.h @@ -0,0 +1,95 @@ +#ifndef Py_CPYTHON_LONGOBJECT_H +# error "this header file must not be included directly" +#endif + +PyAPI_FUNC(int) _PyLong_AsInt(PyObject *); + +PyAPI_FUNC(int) _PyLong_UnsignedShort_Converter(PyObject *, void *); +PyAPI_FUNC(int) _PyLong_UnsignedInt_Converter(PyObject *, void *); +PyAPI_FUNC(int) _PyLong_UnsignedLong_Converter(PyObject *, void *); +PyAPI_FUNC(int) _PyLong_UnsignedLongLong_Converter(PyObject *, void *); +PyAPI_FUNC(int) _PyLong_Size_t_Converter(PyObject *, void *); + +/* _PyLong_Frexp returns a double x and an exponent e such that the + true value is approximately equal to x * 2**e. e is >= 0. x is + 0.0 if and only if the input is 0 (in which case, e and x are both + zeroes); otherwise, 0.5 <= abs(x) < 1.0. On overflow, which is + possible if the number of bits doesn't fit into a Py_ssize_t, sets + OverflowError and returns -1.0 for x, 0 for e. */ +PyAPI_FUNC(double) _PyLong_Frexp(PyLongObject *a, Py_ssize_t *e); + +PyAPI_FUNC(PyObject *) PyLong_FromUnicodeObject(PyObject *u, int base); +PyAPI_FUNC(PyObject *) _PyLong_FromBytes(const char *, Py_ssize_t, int); + +/* _PyLong_Sign. Return 0 if v is 0, -1 if v < 0, +1 if v > 0. + v must not be NULL, and must be a normalized long. + There are no error cases. +*/ +PyAPI_FUNC(int) _PyLong_Sign(PyObject *v); + +/* _PyLong_NumBits. Return the number of bits needed to represent the + absolute value of a long. For example, this returns 1 for 1 and -1, 2 + for 2 and -2, and 2 for 3 and -3. It returns 0 for 0. + v must not be NULL, and must be a normalized long. + (size_t)-1 is returned and OverflowError set if the true result doesn't + fit in a size_t. +*/ +PyAPI_FUNC(size_t) _PyLong_NumBits(PyObject *v); + +/* _PyLong_DivmodNear. Given integers a and b, compute the nearest + integer q to the exact quotient a / b, rounding to the nearest even integer + in the case of a tie. Return (q, r), where r = a - q*b. The remainder r + will satisfy abs(r) <= abs(b)/2, with equality possible only if q is + even. +*/ +PyAPI_FUNC(PyObject *) _PyLong_DivmodNear(PyObject *, PyObject *); + +/* _PyLong_FromByteArray: View the n unsigned bytes as a binary integer in + base 256, and return a Python int with the same numeric value. + If n is 0, the integer is 0. Else: + If little_endian is 1/true, bytes[n-1] is the MSB and bytes[0] the LSB; + else (little_endian is 0/false) bytes[0] is the MSB and bytes[n-1] the + LSB. + If is_signed is 0/false, view the bytes as a non-negative integer. + If is_signed is 1/true, view the bytes as a 2's-complement integer, + non-negative if bit 0x80 of the MSB is clear, negative if set. + Error returns: + + Return NULL with the appropriate exception set if there's not + enough memory to create the Python int. +*/ +PyAPI_FUNC(PyObject *) _PyLong_FromByteArray( + const unsigned char* bytes, size_t n, + int little_endian, int is_signed); + +/* _PyLong_AsByteArray: Convert the least-significant 8*n bits of long + v to a base-256 integer, stored in array bytes. Normally return 0, + return -1 on error. + If little_endian is 1/true, store the MSB at bytes[n-1] and the LSB at + bytes[0]; else (little_endian is 0/false) store the MSB at bytes[0] and + the LSB at bytes[n-1]. + If is_signed is 0/false, it's an error if v < 0; else (v >= 0) n bytes + are filled and there's nothing special about bit 0x80 of the MSB. + If is_signed is 1/true, bytes is filled with the 2's-complement + representation of v's value. Bit 0x80 of the MSB is the sign bit. + Error returns (-1): + + is_signed is 0 and v < 0. TypeError is set in this case, and bytes + isn't altered. + + n isn't big enough to hold the full mathematical value of v. For + example, if is_signed is 0 and there are more digits in the v than + fit in n; or if is_signed is 1, v < 0, and n is just 1 bit shy of + being large enough to hold a sign bit. OverflowError is set in this + case, but bytes holds the least-significant n bytes of the true value. +*/ +PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v, + unsigned char* bytes, size_t n, + int little_endian, int is_signed); + +/* _PyLong_Format: Convert the long to a string object with given base, + appending a base prefix of 0[box] if base is 2, 8 or 16. */ +PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *obj, int base); + +/* For use by the gcd function in mathmodule.c */ +PyAPI_FUNC(PyObject *) _PyLong_GCD(PyObject *, PyObject *); + +PyAPI_FUNC(PyObject *) _PyLong_Rshift(PyObject *, size_t); +PyAPI_FUNC(PyObject *) _PyLong_Lshift(PyObject *, size_t); diff --git a/Include/internal/pycore_long.h b/Include/internal/pycore_long.h index 8edc90218547f..8bdf8e5736d20 100644 --- a/Include/internal/pycore_long.h +++ b/Include/internal/pycore_long.h @@ -37,6 +37,32 @@ static inline PyObject* _PyLong_GetOne(void) PyObject *_PyLong_Add(PyLongObject *left, PyLongObject *right); PyObject *_PyLong_Multiply(PyLongObject *left, PyLongObject *right); +/* Used by Python/mystrtoul.c, _PyBytes_FromHex(), + _PyBytes_DecodeEscape(), etc. */ +PyAPI_DATA(unsigned char) _PyLong_DigitValue[256]; + +/* Format the object based on the format_spec, as defined in PEP 3101 + (Advanced String Formatting). */ +PyAPI_FUNC(int) _PyLong_FormatAdvancedWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + PyObject *format_spec, + Py_ssize_t start, + Py_ssize_t end); + +PyAPI_FUNC(int) _PyLong_FormatWriter( + _PyUnicodeWriter *writer, + PyObject *obj, + int base, + int alternate); + +PyAPI_FUNC(char*) _PyLong_FormatBytesWriter( + _PyBytesWriter *writer, + char *str, + PyObject *obj, + int base, + int alternate); + #ifdef __cplusplus } #endif diff --git a/Include/longobject.h b/Include/longobject.h index e2301d7abfccc..7fe8f58cb3e95 100644 --- a/Include/longobject.h +++ b/Include/longobject.h @@ -26,9 +26,6 @@ PyAPI_FUNC(Py_ssize_t) PyLong_AsSsize_t(PyObject *); PyAPI_FUNC(size_t) PyLong_AsSize_t(PyObject *); PyAPI_FUNC(unsigned long) PyLong_AsUnsignedLong(PyObject *); PyAPI_FUNC(unsigned long) PyLong_AsUnsignedLongMask(PyObject *); -#ifndef Py_LIMITED_API -PyAPI_FUNC(int) _PyLong_AsInt(PyObject *); -#endif PyAPI_FUNC(PyObject *) PyLong_GetInfo(void); /* It may be useful in the future. I've added it in the PyInt -> PyLong @@ -65,30 +62,6 @@ PyAPI_FUNC(PyObject *) PyLong_GetInfo(void); # error "void* different in size from int, long and long long" #endif /* SIZEOF_VOID_P */ -#ifndef Py_LIMITED_API -PyAPI_FUNC(int) _PyLong_UnsignedShort_Converter(PyObject *, void *); -PyAPI_FUNC(int) _PyLong_UnsignedInt_Converter(PyObject *, void *); -PyAPI_FUNC(int) _PyLong_UnsignedLong_Converter(PyObject *, void *); -PyAPI_FUNC(int) _PyLong_UnsignedLongLong_Converter(PyObject *, void *); -PyAPI_FUNC(int) _PyLong_Size_t_Converter(PyObject *, void *); -#endif - -/* Used by Python/mystrtoul.c, _PyBytes_FromHex(), - _PyBytes_DecodeEscape(), etc. */ -#ifndef Py_LIMITED_API -PyAPI_DATA(unsigned char) _PyLong_DigitValue[256]; -#endif - -/* _PyLong_Frexp returns a double x and an exponent e such that the - true value is approximately equal to x * 2**e. e is >= 0. x is - 0.0 if and only if the input is 0 (in which case, e and x are both - zeroes); otherwise, 0.5 <= abs(x) < 1.0. On overflow, which is - possible if the number of bits doesn't fit into a Py_ssize_t, sets - OverflowError and returns -1.0 for x, 0 for e. */ -#ifndef Py_LIMITED_API -PyAPI_FUNC(double) _PyLong_Frexp(PyLongObject *a, Py_ssize_t *e); -#endif - PyAPI_FUNC(double) PyLong_AsDouble(PyObject *); PyAPI_FUNC(PyObject *) PyLong_FromVoidPtr(void *); PyAPI_FUNC(void *) PyLong_AsVoidPtr(PyObject *); @@ -101,102 +74,6 @@ PyAPI_FUNC(unsigned long long) PyLong_AsUnsignedLongLongMask(PyObject *); PyAPI_FUNC(long long) PyLong_AsLongLongAndOverflow(PyObject *, int *); PyAPI_FUNC(PyObject *) PyLong_FromString(const char *, char **, int); -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) PyLong_FromUnicodeObject(PyObject *u, int base); -PyAPI_FUNC(PyObject *) _PyLong_FromBytes(const char *, Py_ssize_t, int); -#endif - -#ifndef Py_LIMITED_API -/* _PyLong_Sign. Return 0 if v is 0, -1 if v < 0, +1 if v > 0. - v must not be NULL, and must be a normalized long. - There are no error cases. -*/ -PyAPI_FUNC(int) _PyLong_Sign(PyObject *v); - - -/* _PyLong_NumBits. Return the number of bits needed to represent the - absolute value of a long. For example, this returns 1 for 1 and -1, 2 - for 2 and -2, and 2 for 3 and -3. It returns 0 for 0. - v must not be NULL, and must be a normalized long. - (size_t)-1 is returned and OverflowError set if the true result doesn't - fit in a size_t. -*/ -PyAPI_FUNC(size_t) _PyLong_NumBits(PyObject *v); - -/* _PyLong_DivmodNear. Given integers a and b, compute the nearest - integer q to the exact quotient a / b, rounding to the nearest even integer - in the case of a tie. Return (q, r), where r = a - q*b. The remainder r - will satisfy abs(r) <= abs(b)/2, with equality possible only if q is - even. -*/ -PyAPI_FUNC(PyObject *) _PyLong_DivmodNear(PyObject *, PyObject *); - -/* _PyLong_FromByteArray: View the n unsigned bytes as a binary integer in - base 256, and return a Python int with the same numeric value. - If n is 0, the integer is 0. Else: - If little_endian is 1/true, bytes[n-1] is the MSB and bytes[0] the LSB; - else (little_endian is 0/false) bytes[0] is the MSB and bytes[n-1] the - LSB. - If is_signed is 0/false, view the bytes as a non-negative integer. - If is_signed is 1/true, view the bytes as a 2's-complement integer, - non-negative if bit 0x80 of the MSB is clear, negative if set. - Error returns: - + Return NULL with the appropriate exception set if there's not - enough memory to create the Python int. -*/ -PyAPI_FUNC(PyObject *) _PyLong_FromByteArray( - const unsigned char* bytes, size_t n, - int little_endian, int is_signed); - -/* _PyLong_AsByteArray: Convert the least-significant 8*n bits of long - v to a base-256 integer, stored in array bytes. Normally return 0, - return -1 on error. - If little_endian is 1/true, store the MSB at bytes[n-1] and the LSB at - bytes[0]; else (little_endian is 0/false) store the MSB at bytes[0] and - the LSB at bytes[n-1]. - If is_signed is 0/false, it's an error if v < 0; else (v >= 0) n bytes - are filled and there's nothing special about bit 0x80 of the MSB. - If is_signed is 1/true, bytes is filled with the 2's-complement - representation of v's value. Bit 0x80 of the MSB is the sign bit. - Error returns (-1): - + is_signed is 0 and v < 0. TypeError is set in this case, and bytes - isn't altered. - + n isn't big enough to hold the full mathematical value of v. For - example, if is_signed is 0 and there are more digits in the v than - fit in n; or if is_signed is 1, v < 0, and n is just 1 bit shy of - being large enough to hold a sign bit. OverflowError is set in this - case, but bytes holds the least-significant n bytes of the true value. -*/ -PyAPI_FUNC(int) _PyLong_AsByteArray(PyLongObject* v, - unsigned char* bytes, size_t n, - int little_endian, int is_signed); - -/* _PyLong_Format: Convert the long to a string object with given base, - appending a base prefix of 0[box] if base is 2, 8 or 16. */ -PyAPI_FUNC(PyObject *) _PyLong_Format(PyObject *obj, int base); - -PyAPI_FUNC(int) _PyLong_FormatWriter( - _PyUnicodeWriter *writer, - PyObject *obj, - int base, - int alternate); - -PyAPI_FUNC(char*) _PyLong_FormatBytesWriter( - _PyBytesWriter *writer, - char *str, - PyObject *obj, - int base, - int alternate); - -/* Format the object based on the format_spec, as defined in PEP 3101 - (Advanced String Formatting). */ -PyAPI_FUNC(int) _PyLong_FormatAdvancedWriter( - _PyUnicodeWriter *writer, - PyObject *obj, - PyObject *format_spec, - Py_ssize_t start, - Py_ssize_t end); -#endif /* Py_LIMITED_API */ /* These aren't really part of the int object, but they're handy. The functions are in Python/mystrtoul.c. @@ -205,13 +82,9 @@ PyAPI_FUNC(unsigned long) PyOS_strtoul(const char *, char **, int); PyAPI_FUNC(long) PyOS_strtol(const char *, char **, int); #ifndef Py_LIMITED_API -/* For use by the gcd function in mathmodule.c */ -PyAPI_FUNC(PyObject *) _PyLong_GCD(PyObject *, PyObject *); -#endif /* !Py_LIMITED_API */ - -#ifndef Py_LIMITED_API -PyAPI_FUNC(PyObject *) _PyLong_Rshift(PyObject *, size_t); -PyAPI_FUNC(PyObject *) _PyLong_Lshift(PyObject *, size_t); +# define Py_CPYTHON_LONGOBJECT_H +# include "cpython/longobject.h" +# undef Py_CPYTHON_LONGOBJECT_H #endif #ifdef __cplusplus diff --git a/Makefile.pre.in b/Makefile.pre.in index 60dc22489dd66..f03f535f6faa6 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1210,6 +1210,7 @@ PYTHON_HEADERS= \ $(srcdir)/Include/cpython/initconfig.h \ $(srcdir)/Include/cpython/listobject.h \ $(srcdir)/Include/cpython/longintrepr.h \ + $(srcdir)/Include/cpython/longobject.h \ $(srcdir)/Include/cpython/methodobject.h \ $(srcdir)/Include/cpython/object.h \ $(srcdir)/Include/cpython/objimpl.h \ diff --git a/Modules/binascii.c b/Modules/binascii.c index ef209881494ac..7037d34dbe2dd 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -56,6 +56,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" +#include "pycore_long.h" // _PyLong_DigitValue #include "pycore_strhex.h" // _Py_strhex_bytes_with_sep() #ifdef USE_ZLIB_CRC32 # include "zlib.h" diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 1163cf00034b2..66fd2ecc3c41d 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -8,6 +8,7 @@ #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_format.h" // F_LJUST #include "pycore_initconfig.h" // _PyStatus_OK() +#include "pycore_long.h" // _PyLong_DigitValue #include "pycore_object.h" // _PyObject_GC_TRACK #include "pycore_pymem.h" // PYMEM_CLEANBYTE #include "pycore_strhex.h" // _Py_strhex_with_sep() diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index 386052f31bea2..61fc34d71da3c 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -46,6 +46,7 @@ OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. #include "pycore_format.h" // F_LJUST #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_interp.h" // PyInterpreterState.fs_codec +#include "pycore_long.h" // _PyLong_FormatWriter() #include "pycore_object.h" // _PyObject_GC_TRACK() #include "pycore_pathconfig.h" // _Py_DumpPathConfig() #include "pycore_pyerrors.h" // _Py_FatalRefcountError() diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index a26ba9c232e41..1b484be5e7a2f 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -140,6 +140,7 @@ + diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index 335bfb31bed54..c1519760d2207 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -387,6 +387,9 @@ Include + + Include + Include diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c index 9071bf3a6589e..0c21301fb9ed4 100644 --- a/Python/formatter_unicode.c +++ b/Python/formatter_unicode.c @@ -3,7 +3,8 @@ of int.__float__, etc., that take and return unicode objects */ #include "Python.h" -#include "pycore_fileutils.h" +#include "pycore_fileutils.h" // _Py_GetLocaleconvNumeric() +#include "pycore_long.h" // _PyLong_FormatWriter() #include /* Raises an exception about an unknown presentation type for this diff --git a/Python/mystrtoul.c b/Python/mystrtoul.c index 19fa57aa144e2..e6fe154eed611 100644 --- a/Python/mystrtoul.c +++ b/Python/mystrtoul.c @@ -1,5 +1,5 @@ - #include "Python.h" +#include "pycore_long.h" // _PyLong_DigitValue #if defined(__sgi) && !defined(_SGI_MP_SOURCE) #define _SGI_MP_SOURCE From webhook-mailer at python.org Mon Oct 18 20:52:37 2021 From: webhook-mailer at python.org (jaraco) Date: Tue, 19 Oct 2021 00:52:37 -0000 Subject: [Python-checkins] bpo-45516: use documentation links in TraversableResources' description (GH-29045) Message-ID: https://github.com/python/cpython/commit/c0295675305f6896e4ba7496441cc470d7edca89 commit: c0295675305f6896e4ba7496441cc470d7edca89 branch: main author: Filipe La?ns committer: jaraco date: 2021-10-18T20:52:28-04:00 summary: bpo-45516: use documentation links in TraversableResources' description (GH-29045) I think this makes the documentation much more digestible :) Signed-off-by: Filipe La?ns files: M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 3576941efa46d..19230570f2d5f 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -859,10 +859,11 @@ ABC hierarchy:: .. class:: TraversableResources An abstract base class for resource readers capable of serving - the ``files`` interface. Subclasses ResourceReader and provides - concrete implementations of the ResourceReader's abstract - methods. Therefore, any loader supplying TraversableReader - also supplies ResourceReader. + the :meth:`importlib.resources.files` interface. Subclasses + :class:`importlib.abc.ResourceReader` and provides + concrete implementations of the :class:`importlib.abc.ResourceReader`'s + abstract methods. Therefore, any loader supplying + :class:`importlib.abc.TraversableReader` also supplies ResourceReader. Loaders that wish to support resource reading are expected to implement this interface. From webhook-mailer at python.org Mon Oct 18 22:52:01 2021 From: webhook-mailer at python.org (methane) Date: Tue, 19 Oct 2021 02:52:01 -0000 Subject: [Python-checkins] bpo-45475: Revert `__iter__` optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016) Message-ID: https://github.com/python/cpython/commit/0a4c82ddd34a3578684b45b76f49cd289a08740b commit: 0a4c82ddd34a3578684b45b76f49cd289a08740b branch: main author: Inada Naoki committer: methane date: 2021-10-19T11:51:48+09:00 summary: bpo-45475: Revert `__iter__` optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016) This reverts commit d2a8e69c2c605fbaa3656a5f99aa8d295f74c80e. files: A Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst M Lib/bz2.py M Lib/gzip.py M Lib/lzma.py diff --git a/Lib/bz2.py b/Lib/bz2.py index 7f1d20632ef13..fabe4f73c8d80 100644 --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -197,10 +197,6 @@ def readline(self, size=-1): self._check_can_read() return self._buffer.readline(size) - def __iter__(self): - self._check_can_read() - return self._buffer.__iter__() - def readlines(self, size=-1): """Read a list of lines of uncompressed bytes from the file. diff --git a/Lib/gzip.py b/Lib/gzip.py index 0dddb51553fab..ac1781042b264 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -398,10 +398,6 @@ def readline(self, size=-1): self._check_not_closed() return self._buffer.readline(size) - def __iter__(self): - self._check_not_closed() - return self._buffer.__iter__() - def _read_exact(fp, n): '''Read exactly *n* bytes from `fp` diff --git a/Lib/lzma.py b/Lib/lzma.py index 9abf06d91db18..800f52198fbb7 100644 --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -221,10 +221,6 @@ def readline(self, size=-1): self._check_can_read() return self._buffer.readline(size) - def __iter__(self): - self._check_can_read() - return self._buffer.__iter__() - def write(self, data): """Write a bytes object to the file. diff --git a/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst b/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst new file mode 100644 index 0000000000000..6fce894e6e4d4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst @@ -0,0 +1,4 @@ +Reverted optimization of iterating :class:`gzip.GzipFile`, +:class:`bz2.BZ2File`, and :class:`lzma.LZMAFile` (see bpo-43787) because it +caused regression when user iterate them without having reference of them. +Patch by Inada Naoki. From webhook-mailer at python.org Mon Oct 18 23:15:56 2021 From: webhook-mailer at python.org (miss-islington) Date: Tue, 19 Oct 2021 03:15:56 -0000 Subject: [Python-checkins] bpo-45475: Revert `__iter__` optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016) Message-ID: https://github.com/python/cpython/commit/97ce855ca8ce437070424b43f5b41158685ac140 commit: 97ce855ca8ce437070424b43f5b41158685ac140 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-18T20:15:48-07:00 summary: bpo-45475: Revert `__iter__` optimization for GzipFile, BZ2File, and LZMAFile. (GH-29016) This reverts commit d2a8e69c2c605fbaa3656a5f99aa8d295f74c80e. (cherry picked from commit 0a4c82ddd34a3578684b45b76f49cd289a08740b) Co-authored-by: Inada Naoki files: A Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst M Lib/bz2.py M Lib/gzip.py M Lib/lzma.py diff --git a/Lib/bz2.py b/Lib/bz2.py index 7f1d20632ef13..fabe4f73c8d80 100644 --- a/Lib/bz2.py +++ b/Lib/bz2.py @@ -197,10 +197,6 @@ def readline(self, size=-1): self._check_can_read() return self._buffer.readline(size) - def __iter__(self): - self._check_can_read() - return self._buffer.__iter__() - def readlines(self, size=-1): """Read a list of lines of uncompressed bytes from the file. diff --git a/Lib/gzip.py b/Lib/gzip.py index 3d837b744800e..475ec326c0c98 100644 --- a/Lib/gzip.py +++ b/Lib/gzip.py @@ -398,10 +398,6 @@ def readline(self, size=-1): self._check_not_closed() return self._buffer.readline(size) - def __iter__(self): - self._check_not_closed() - return self._buffer.__iter__() - class _GzipReader(_compression.DecompressReader): def __init__(self, fp): diff --git a/Lib/lzma.py b/Lib/lzma.py index 9abf06d91db18..800f52198fbb7 100644 --- a/Lib/lzma.py +++ b/Lib/lzma.py @@ -221,10 +221,6 @@ def readline(self, size=-1): self._check_can_read() return self._buffer.readline(size) - def __iter__(self): - self._check_can_read() - return self._buffer.__iter__() - def write(self, data): """Write a bytes object to the file. diff --git a/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst b/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst new file mode 100644 index 0000000000000..6fce894e6e4d4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-10-46-47.bpo-45475.sb9KDF.rst @@ -0,0 +1,4 @@ +Reverted optimization of iterating :class:`gzip.GzipFile`, +:class:`bz2.BZ2File`, and :class:`lzma.LZMAFile` (see bpo-43787) because it +caused regression when user iterate them without having reference of them. +Patch by Inada Naoki. From webhook-mailer at python.org Tue Oct 19 06:10:27 2021 From: webhook-mailer at python.org (vstinner) Date: Tue, 19 Oct 2021 10:10:27 -0000 Subject: [Python-checkins] bpo-45434: Only exclude in Python 3.11 limited C API (GH-29027) Message-ID: https://github.com/python/cpython/commit/52af0756b2ffc6788e364971d05cdaf127d77d5a commit: 52af0756b2ffc6788e364971d05cdaf127d77d5a branch: main author: Victor Stinner committer: vstinner date: 2021-10-19T12:10:22+02:00 summary: bpo-45434: Only exclude in Python 3.11 limited C API (GH-29027) The Python 3.11 limited C API no longer includes stdlib.h, stdio.h, string.h and errno.h. * Exclude Py_MEMCPY() from Python 3.11 limited C API. * xxlimited C extension is now built with Python 3.11 limited C API. files: D Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst M Doc/whatsnew/3.11.rst M Include/Python.h M Include/pyport.h M Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst M Modules/xxlimited.c diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 0776265779849..2e57f0cea53db 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -564,18 +564,10 @@ Porting to Python 3.11 (Contributed by Victor Stinner in :issue:`39573`.) -* The ```` header file no longer includes ````. C - extensions using ```` must now include it explicitly. - The system ```` header provides functions like: - ``malloc()``/``free()``, ``getenv()``, ``strtol()``, ``abs()``, ``strtol()``, - ``exit()`` and ``abort()``. - (Contributed by Victor Stinner in :issue:`45434`.) - -* The ```` header file no longer includes ```` if the - ``Py_LIMITED_API`` macro is defined. Functions expecting ``FILE*`` are - excluded from the limited C API (:pep:`384`). C extensions using - ```` must now include it explicitly. The system ```` - header provides functions like ``printf()`` and ``fopen()``. +* ```` no longer includes the header files ````, + ````, ```` and ```` when the ``Py_LIMITED_API`` + macro is set to ``0x030b0000`` (Python 3.11) or higher. C extensions should + explicitly include the header files after ``#include ``. (Contributed by Victor Stinner in :issue:`45434`.) * The non-limited API files ``cellobject.h``, ``classobject.h``, ``context.h``, diff --git a/Include/Python.h b/Include/Python.h index c0a621ad44afd..6e3303ac9a3b0 100644 --- a/Include/Python.h +++ b/Include/Python.h @@ -16,12 +16,14 @@ # define _SGI_MP_SOURCE #endif -#include // memcpy() -#ifndef Py_LIMITED_API +// stdlib.h, stdio.h, errno.h and string.h headers are not used by Python +// headers, but kept for backward compatibility. They are excluded from the +// limited C API of Python 3.11. +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# include # include // FILE* -#endif -#ifdef HAVE_ERRNO_H # include // errno +# include // memcpy() #endif #ifndef MS_WINDOWS # include diff --git a/Include/pyport.h b/Include/pyport.h index 0bec2a9b38f7c..61ca3a97c186b 100644 --- a/Include/pyport.h +++ b/Include/pyport.h @@ -201,9 +201,10 @@ typedef Py_ssize_t Py_ssize_clean_t; # define Py_LOCAL_INLINE(type) static inline type #endif -/* Py_MEMCPY is kept for backwards compatibility, - * see https://bugs.python.org/issue28126 */ -#define Py_MEMCPY memcpy +// bpo-28126: Py_MEMCPY is kept for backwards compatibility, +#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 < 0x030b0000 +# define Py_MEMCPY memcpy +#endif #ifdef HAVE_IEEEFP_H #include /* needed for 'finite' declaration on some platforms */ diff --git a/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst index 95c5f0d1150c8..627f687272f5a 100644 --- a/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst +++ b/Misc/NEWS.d/next/C API/2021-10-11-23-03-49.bpo-45434.tsS8I_.rst @@ -1,6 +1,5 @@ -The ```` header file no longer includes ````. C -extensions using ```` must now include it explicitly. -The system ```` header provides functions like: -``malloc()``/``free()``, ``getenv()``, ``strtol()``, ``abs()``, ``strtol()``, -``exit()`` and ``abort()``. +```` no longer includes the header files ````, +````, ```` and ```` when the ``Py_LIMITED_API`` +macro is set to ``0x030b0000`` (Python 3.11) or higher. C extensions should +explicitly include the header files after ``#include ``. Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst b/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst deleted file mode 100644 index 4a06635d179d9..0000000000000 --- a/Misc/NEWS.d/next/C API/2021-10-15-00-30-45.bpo-45434.XLtsbK.rst +++ /dev/null @@ -1,5 +0,0 @@ -The ```` header file no longer includes ```` if the -``Py_LIMITED_API`` macro is defined. Functions expecting ``FILE*`` are excluded -from the limited C API (:pep:`384`). C extensions using ```` must now -include it explicitly. -Patch by Victor Stinner. diff --git a/Modules/xxlimited.c b/Modules/xxlimited.c index 9bd9a5407d43c..93895c4f1214c 100644 --- a/Modules/xxlimited.c +++ b/Modules/xxlimited.c @@ -1,4 +1,3 @@ - /* Use this file as a template to start implementing a module that also declares object types. All occurrences of 'Xxo' should be changed to something reasonable for your objects. After that, all other @@ -55,7 +54,7 @@ pass */ -#define Py_LIMITED_API 0x030a0000 +#define Py_LIMITED_API 0x030b0000 #include "Python.h" From webhook-mailer at python.org Tue Oct 19 07:00:02 2021 From: webhook-mailer at python.org (zooba) Date: Tue, 19 Oct 2021 11:00:02 -0000 Subject: [Python-checkins] bpo-43851: Build SQLite with SQLITE_OMIT_AUTOINIT on Windows (GH-25414) Message-ID: https://github.com/python/cpython/commit/8702b667d8e7da7a1888297d84f493f26c580a2d commit: 8702b667d8e7da7a1888297d84f493f26c580a2d branch: main author: Erlend Egeberg Aasland committer: zooba date: 2021-10-19T11:59:57+01:00 summary: bpo-43851: Build SQLite with SQLITE_OMIT_AUTOINIT on Windows (GH-25414) files: A Misc/NEWS.d/next/Windows/2021-04-15-01-23-10.bpo-43851.qgU0gy.rst M PCbuild/sqlite3.vcxproj diff --git a/Misc/NEWS.d/next/Windows/2021-04-15-01-23-10.bpo-43851.qgU0gy.rst b/Misc/NEWS.d/next/Windows/2021-04-15-01-23-10.bpo-43851.qgU0gy.rst new file mode 100644 index 0000000000000..e9555d56754cc --- /dev/null +++ b/Misc/NEWS.d/next/Windows/2021-04-15-01-23-10.bpo-43851.qgU0gy.rst @@ -0,0 +1 @@ +Build SQLite ``SQLITE_OMIT_AUTOINIT`` on Windows. Patch by Erlend E. Aasland. diff --git a/PCbuild/sqlite3.vcxproj b/PCbuild/sqlite3.vcxproj index e39e2d9c226ca..310795c4e4d20 100644 --- a/PCbuild/sqlite3.vcxproj +++ b/PCbuild/sqlite3.vcxproj @@ -98,7 +98,7 @@ $(sqlite3Dir);%(AdditionalIncludeDirectories) - SQLITE_ENABLE_MATH_FUNCTIONS;SQLITE_ENABLE_JSON1;SQLITE_ENABLE_FTS4;SQLITE_ENABLE_FTS5;SQLITE_ENABLE_RTREE;SQLITE_API=__declspec(dllexport);%(PreprocessorDefinitions) + SQLITE_ENABLE_MATH_FUNCTIONS;SQLITE_ENABLE_JSON1;SQLITE_ENABLE_FTS4;SQLITE_ENABLE_FTS5;SQLITE_ENABLE_RTREE;SQLITE_OMIT_AUTOINIT;SQLITE_API=__declspec(dllexport);%(PreprocessorDefinitions) Level1 From webhook-mailer at python.org Tue Oct 19 09:44:55 2021 From: webhook-mailer at python.org (encukou) Date: Tue, 19 Oct 2021 13:44:55 -0000 Subject: [Python-checkins] bpo-42064: Add module backref to `sqlite3` callback context (GH-28242) Message-ID: https://github.com/python/cpython/commit/09c04e7f0d26f0006964554b6a0caa5ef7f0bd24 commit: 09c04e7f0d26f0006964554b6a0caa5ef7f0bd24 branch: main author: Erlend Egeberg Aasland committer: encukou date: 2021-10-19T15:44:45+02:00 summary: bpo-42064: Add module backref to `sqlite3` callback context (GH-28242) files: M Modules/_sqlite/clinic/connection.c.h M Modules/_sqlite/connection.c M Modules/_sqlite/connection.h diff --git a/Modules/_sqlite/clinic/connection.c.h b/Modules/_sqlite/clinic/connection.c.h index bf5a58d7756c9..e9e3064ae0f89 100644 --- a/Modules/_sqlite/clinic/connection.c.h +++ b/Modules/_sqlite/clinic/connection.c.h @@ -204,57 +204,30 @@ PyDoc_STRVAR(pysqlite_connection_create_function__doc__, "Creates a new function. Non-standard."); #define PYSQLITE_CONNECTION_CREATE_FUNCTION_METHODDEF \ - {"create_function", (PyCFunction)(void(*)(void))pysqlite_connection_create_function, METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_create_function__doc__}, + {"create_function", (PyCFunction)(void(*)(void))pysqlite_connection_create_function, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_create_function__doc__}, static PyObject * pysqlite_connection_create_function_impl(pysqlite_Connection *self, - const char *name, int narg, - PyObject *func, int deterministic); + PyTypeObject *cls, const char *name, + int narg, PyObject *func, + int deterministic); static PyObject * -pysqlite_connection_create_function(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_connection_create_function(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; static const char * const _keywords[] = {"name", "narg", "func", "deterministic", NULL}; - static _PyArg_Parser _parser = {NULL, _keywords, "create_function", 0}; - PyObject *argsbuf[4]; - Py_ssize_t noptargs = nargs + (kwnames ? PyTuple_GET_SIZE(kwnames) : 0) - 3; + static _PyArg_Parser _parser = {"siO|$p:create_function", _keywords, 0}; const char *name; int narg; PyObject *func; int deterministic = 0; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); - if (!args) { - goto exit; - } - if (!PyUnicode_Check(args[0])) { - _PyArg_BadArgument("create_function", "argument 'name'", "str", args[0]); - goto exit; - } - Py_ssize_t name_length; - name = PyUnicode_AsUTF8AndSize(args[0], &name_length); - if (name == NULL) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &name, &narg, &func, &deterministic)) { goto exit; } - if (strlen(name) != (size_t)name_length) { - PyErr_SetString(PyExc_ValueError, "embedded null character"); - goto exit; - } - narg = _PyLong_AsInt(args[1]); - if (narg == -1 && PyErr_Occurred()) { - goto exit; - } - func = args[2]; - if (!noptargs) { - goto skip_optional_kwonly; - } - deterministic = PyObject_IsTrue(args[3]); - if (deterministic < 0) { - goto exit; - } -skip_optional_kwonly: - return_value = pysqlite_connection_create_function_impl(self, name, narg, func, deterministic); + return_value = pysqlite_connection_create_function_impl(self, cls, name, narg, func, deterministic); exit: return return_value; @@ -267,47 +240,29 @@ PyDoc_STRVAR(pysqlite_connection_create_aggregate__doc__, "Creates a new aggregate. Non-standard."); #define PYSQLITE_CONNECTION_CREATE_AGGREGATE_METHODDEF \ - {"create_aggregate", (PyCFunction)(void(*)(void))pysqlite_connection_create_aggregate, METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_create_aggregate__doc__}, + {"create_aggregate", (PyCFunction)(void(*)(void))pysqlite_connection_create_aggregate, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_create_aggregate__doc__}, static PyObject * pysqlite_connection_create_aggregate_impl(pysqlite_Connection *self, + PyTypeObject *cls, const char *name, int n_arg, PyObject *aggregate_class); static PyObject * -pysqlite_connection_create_aggregate(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_connection_create_aggregate(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; static const char * const _keywords[] = {"name", "n_arg", "aggregate_class", NULL}; - static _PyArg_Parser _parser = {NULL, _keywords, "create_aggregate", 0}; - PyObject *argsbuf[3]; + static _PyArg_Parser _parser = {"siO:create_aggregate", _keywords, 0}; const char *name; int n_arg; PyObject *aggregate_class; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 3, 3, 0, argsbuf); - if (!args) { - goto exit; - } - if (!PyUnicode_Check(args[0])) { - _PyArg_BadArgument("create_aggregate", "argument 'name'", "str", args[0]); - goto exit; - } - Py_ssize_t name_length; - name = PyUnicode_AsUTF8AndSize(args[0], &name_length); - if (name == NULL) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &name, &n_arg, &aggregate_class)) { goto exit; } - if (strlen(name) != (size_t)name_length) { - PyErr_SetString(PyExc_ValueError, "embedded null character"); - goto exit; - } - n_arg = _PyLong_AsInt(args[1]); - if (n_arg == -1 && PyErr_Occurred()) { - goto exit; - } - aggregate_class = args[2]; - return_value = pysqlite_connection_create_aggregate_impl(self, name, n_arg, aggregate_class); + return_value = pysqlite_connection_create_aggregate_impl(self, cls, name, n_arg, aggregate_class); exit: return return_value; @@ -320,27 +275,26 @@ PyDoc_STRVAR(pysqlite_connection_set_authorizer__doc__, "Sets authorizer callback. Non-standard."); #define PYSQLITE_CONNECTION_SET_AUTHORIZER_METHODDEF \ - {"set_authorizer", (PyCFunction)(void(*)(void))pysqlite_connection_set_authorizer, METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_authorizer__doc__}, + {"set_authorizer", (PyCFunction)(void(*)(void))pysqlite_connection_set_authorizer, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_authorizer__doc__}, static PyObject * pysqlite_connection_set_authorizer_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable); static PyObject * -pysqlite_connection_set_authorizer(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_connection_set_authorizer(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; static const char * const _keywords[] = {"authorizer_callback", NULL}; - static _PyArg_Parser _parser = {NULL, _keywords, "set_authorizer", 0}; - PyObject *argsbuf[1]; + static _PyArg_Parser _parser = {"O:set_authorizer", _keywords, 0}; PyObject *callable; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); - if (!args) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &callable)) { goto exit; } - callable = args[0]; - return_value = pysqlite_connection_set_authorizer_impl(self, callable); + return_value = pysqlite_connection_set_authorizer_impl(self, cls, callable); exit: return return_value; @@ -353,32 +307,27 @@ PyDoc_STRVAR(pysqlite_connection_set_progress_handler__doc__, "Sets progress handler callback. Non-standard."); #define PYSQLITE_CONNECTION_SET_PROGRESS_HANDLER_METHODDEF \ - {"set_progress_handler", (PyCFunction)(void(*)(void))pysqlite_connection_set_progress_handler, METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_progress_handler__doc__}, + {"set_progress_handler", (PyCFunction)(void(*)(void))pysqlite_connection_set_progress_handler, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_progress_handler__doc__}, static PyObject * pysqlite_connection_set_progress_handler_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable, int n); static PyObject * -pysqlite_connection_set_progress_handler(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_connection_set_progress_handler(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; static const char * const _keywords[] = {"progress_handler", "n", NULL}; - static _PyArg_Parser _parser = {NULL, _keywords, "set_progress_handler", 0}; - PyObject *argsbuf[2]; + static _PyArg_Parser _parser = {"Oi:set_progress_handler", _keywords, 0}; PyObject *callable; int n; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 2, 2, 0, argsbuf); - if (!args) { - goto exit; - } - callable = args[0]; - n = _PyLong_AsInt(args[1]); - if (n == -1 && PyErr_Occurred()) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &callable, &n)) { goto exit; } - return_value = pysqlite_connection_set_progress_handler_impl(self, callable, n); + return_value = pysqlite_connection_set_progress_handler_impl(self, cls, callable, n); exit: return return_value; @@ -393,27 +342,26 @@ PyDoc_STRVAR(pysqlite_connection_set_trace_callback__doc__, "Non-standard."); #define PYSQLITE_CONNECTION_SET_TRACE_CALLBACK_METHODDEF \ - {"set_trace_callback", (PyCFunction)(void(*)(void))pysqlite_connection_set_trace_callback, METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_trace_callback__doc__}, + {"set_trace_callback", (PyCFunction)(void(*)(void))pysqlite_connection_set_trace_callback, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_set_trace_callback__doc__}, static PyObject * pysqlite_connection_set_trace_callback_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable); static PyObject * -pysqlite_connection_set_trace_callback(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_connection_set_trace_callback(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; static const char * const _keywords[] = {"trace_callback", NULL}; - static _PyArg_Parser _parser = {NULL, _keywords, "set_trace_callback", 0}; - PyObject *argsbuf[1]; + static _PyArg_Parser _parser = {"O:set_trace_callback", _keywords, 0}; PyObject *callable; - args = _PyArg_UnpackKeywords(args, nargs, NULL, kwnames, &_parser, 1, 1, 0, argsbuf); - if (!args) { + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &callable)) { goto exit; } - callable = args[0]; - return_value = pysqlite_connection_set_trace_callback_impl(self, callable); + return_value = pysqlite_connection_set_trace_callback_impl(self, cls, callable); exit: return return_value; @@ -720,38 +668,28 @@ PyDoc_STRVAR(pysqlite_connection_create_collation__doc__, "Creates a collation function. Non-standard."); #define PYSQLITE_CONNECTION_CREATE_COLLATION_METHODDEF \ - {"create_collation", (PyCFunction)(void(*)(void))pysqlite_connection_create_collation, METH_FASTCALL, pysqlite_connection_create_collation__doc__}, + {"create_collation", (PyCFunction)(void(*)(void))pysqlite_connection_create_collation, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_connection_create_collation__doc__}, static PyObject * pysqlite_connection_create_collation_impl(pysqlite_Connection *self, + PyTypeObject *cls, const char *name, PyObject *callable); static PyObject * -pysqlite_connection_create_collation(pysqlite_Connection *self, PyObject *const *args, Py_ssize_t nargs) +pysqlite_connection_create_collation(pysqlite_Connection *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) { PyObject *return_value = NULL; + static const char * const _keywords[] = {"", "", NULL}; + static _PyArg_Parser _parser = {"sO:create_collation", _keywords, 0}; const char *name; PyObject *callable; - if (!_PyArg_CheckPositional("create_collation", nargs, 2, 2)) { - goto exit; - } - if (!PyUnicode_Check(args[0])) { - _PyArg_BadArgument("create_collation", "argument 1", "str", args[0]); - goto exit; - } - Py_ssize_t name_length; - name = PyUnicode_AsUTF8AndSize(args[0], &name_length); - if (name == NULL) { - goto exit; - } - if (strlen(name) != (size_t)name_length) { - PyErr_SetString(PyExc_ValueError, "embedded null character"); + if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser, + &name, &callable)) { goto exit; } - callable = args[1]; - return_value = pysqlite_connection_create_collation_impl(self, name, callable); + return_value = pysqlite_connection_create_collation_impl(self, cls, name, callable); exit: return return_value; @@ -819,4 +757,4 @@ pysqlite_connection_exit(pysqlite_Connection *self, PyObject *const *args, Py_ss #ifndef PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF #define PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF #endif /* !defined(PYSQLITE_CONNECTION_LOAD_EXTENSION_METHODDEF) */ -/*[clinic end generated code: output=5b7268875f33c016 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=7567e5d716309258 input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index d6d1fa8bf2876..e94c4cbb4e8c3 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -256,6 +256,7 @@ pysqlite_do_all_statements(pysqlite_Connection *self) do { \ if (ctx) { \ Py_VISIT(ctx->callable); \ + Py_VISIT(ctx->module); \ } \ } while (0) @@ -280,6 +281,7 @@ clear_callback_context(callback_context *ctx) { if (ctx != NULL) { Py_CLEAR(ctx->callable); + Py_CLEAR(ctx->module); } } @@ -822,13 +824,21 @@ static void _pysqlite_drop_unused_cursor_references(pysqlite_Connection* self) Py_SETREF(self->cursors, new_list); } +/* Allocate a UDF/callback context structure. In order to ensure that the state + * pointer always outlives the callback context, we make sure it owns a + * reference to the module itself. create_callback_context() is always called + * from connection methods, so we use the defining class to fetch the module + * pointer. + */ static callback_context * -create_callback_context(pysqlite_state *state, PyObject *callable) +create_callback_context(PyTypeObject *cls, PyObject *callable) { callback_context *ctx = PyMem_Malloc(sizeof(callback_context)); if (ctx != NULL) { + PyObject *module = PyType_GetModule(cls); ctx->callable = Py_NewRef(callable); - ctx->state = state; + ctx->module = Py_NewRef(module); + ctx->state = pysqlite_get_state(module); } return ctx; } @@ -838,6 +848,7 @@ free_callback_context(callback_context *ctx) { assert(ctx != NULL); Py_XDECREF(ctx->callable); + Py_XDECREF(ctx->module); PyMem_Free(ctx); } @@ -867,6 +878,8 @@ destructor_callback(void *ctx) /*[clinic input] _sqlite3.Connection.create_function as pysqlite_connection_create_function + cls: defining_class + / name: str narg: int func: object @@ -878,9 +891,10 @@ Creates a new function. Non-standard. static PyObject * pysqlite_connection_create_function_impl(pysqlite_Connection *self, - const char *name, int narg, - PyObject *func, int deterministic) -/*[clinic end generated code: output=07d1877dd98c0308 input=f2edcf073e815beb]*/ + PyTypeObject *cls, const char *name, + int narg, PyObject *func, + int deterministic) +/*[clinic end generated code: output=8a811529287ad240 input=f0f99754bfeafd8d]*/ { int rc; int flags = SQLITE_UTF8; @@ -903,7 +917,7 @@ pysqlite_connection_create_function_impl(pysqlite_Connection *self, flags |= SQLITE_DETERMINISTIC; #endif } - callback_context *ctx = create_callback_context(self->state, func); + callback_context *ctx = create_callback_context(cls, func); if (ctx == NULL) { return NULL; } @@ -924,6 +938,8 @@ pysqlite_connection_create_function_impl(pysqlite_Connection *self, /*[clinic input] _sqlite3.Connection.create_aggregate as pysqlite_connection_create_aggregate + cls: defining_class + / name: str n_arg: int aggregate_class: object @@ -933,9 +949,10 @@ Creates a new aggregate. Non-standard. static PyObject * pysqlite_connection_create_aggregate_impl(pysqlite_Connection *self, + PyTypeObject *cls, const char *name, int n_arg, PyObject *aggregate_class) -/*[clinic end generated code: output=fbb2f858cfa4d8db input=c2e13bbf234500a5]*/ +/*[clinic end generated code: output=1b02d0f0aec7ff96 input=bd527067e6c2e33f]*/ { int rc; @@ -943,8 +960,7 @@ pysqlite_connection_create_aggregate_impl(pysqlite_Connection *self, return NULL; } - callback_context *ctx = create_callback_context(self->state, - aggregate_class); + callback_context *ctx = create_callback_context(cls, aggregate_class); if (ctx == NULL) { return NULL; } @@ -1075,6 +1091,8 @@ trace_callback(void *ctx, const char *statement_string) /*[clinic input] _sqlite3.Connection.set_authorizer as pysqlite_connection_set_authorizer + cls: defining_class + / authorizer_callback as callable: object Sets authorizer callback. Non-standard. @@ -1082,8 +1100,9 @@ Sets authorizer callback. Non-standard. static PyObject * pysqlite_connection_set_authorizer_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable) -/*[clinic end generated code: output=c193601e9e8a5116 input=ec104f130b82050b]*/ +/*[clinic end generated code: output=75fa60114fc971c3 input=9f3e90d3d642c4a0]*/ { if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { return NULL; @@ -1095,7 +1114,7 @@ pysqlite_connection_set_authorizer_impl(pysqlite_Connection *self, set_callback_context(&self->authorizer_ctx, NULL); } else { - callback_context *ctx = create_callback_context(self->state, callable); + callback_context *ctx = create_callback_context(cls, callable); if (ctx == NULL) { return NULL; } @@ -1114,6 +1133,8 @@ pysqlite_connection_set_authorizer_impl(pysqlite_Connection *self, /*[clinic input] _sqlite3.Connection.set_progress_handler as pysqlite_connection_set_progress_handler + cls: defining_class + / progress_handler as callable: object n: int @@ -1122,8 +1143,9 @@ Sets progress handler callback. Non-standard. static PyObject * pysqlite_connection_set_progress_handler_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable, int n) -/*[clinic end generated code: output=ba14008a483d7a53 input=3cf56d045f130a84]*/ +/*[clinic end generated code: output=0739957fd8034a50 input=83e8dcbb4ce183f7]*/ { if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { return NULL; @@ -1135,7 +1157,7 @@ pysqlite_connection_set_progress_handler_impl(pysqlite_Connection *self, set_callback_context(&self->progress_ctx, NULL); } else { - callback_context *ctx = create_callback_context(self->state, callable); + callback_context *ctx = create_callback_context(cls, callable); if (ctx == NULL) { return NULL; } @@ -1148,6 +1170,8 @@ pysqlite_connection_set_progress_handler_impl(pysqlite_Connection *self, /*[clinic input] _sqlite3.Connection.set_trace_callback as pysqlite_connection_set_trace_callback + cls: defining_class + / trace_callback as callable: object Sets a trace callback called for each SQL statement (passed as unicode). @@ -1157,8 +1181,9 @@ Non-standard. static PyObject * pysqlite_connection_set_trace_callback_impl(pysqlite_Connection *self, + PyTypeObject *cls, PyObject *callable) -/*[clinic end generated code: output=c9fd551e359165d3 input=d76eabbb633057bc]*/ +/*[clinic end generated code: output=d91048c03bfcee05 input=96f03acec3ec8044]*/ { if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { return NULL; @@ -1180,7 +1205,7 @@ pysqlite_connection_set_trace_callback_impl(pysqlite_Connection *self, set_callback_context(&self->trace_ctx, NULL); } else { - callback_context *ctx = create_callback_context(self->state, callable); + callback_context *ctx = create_callback_context(cls, callable); if (ctx == NULL) { return NULL; } @@ -1742,6 +1767,7 @@ pysqlite_connection_backup_impl(pysqlite_Connection *self, /*[clinic input] _sqlite3.Connection.create_collation as pysqlite_connection_create_collation + cls: defining_class name: str callback as callable: object / @@ -1751,9 +1777,10 @@ Creates a collation function. Non-standard. static PyObject * pysqlite_connection_create_collation_impl(pysqlite_Connection *self, + PyTypeObject *cls, const char *name, PyObject *callable) -/*[clinic end generated code: output=a4ceaff957fdef9a input=301647aab0f2fb1d]*/ +/*[clinic end generated code: output=32d339e97869c378 input=fee2c8e5708602ad]*/ { if (!pysqlite_check_thread(self) || !pysqlite_check_connection(self)) { return NULL; @@ -1771,7 +1798,7 @@ pysqlite_connection_create_collation_impl(pysqlite_Connection *self, PyErr_SetString(PyExc_TypeError, "parameter must be callable"); return NULL; } - ctx = create_callback_context(self->state, callable); + ctx = create_callback_context(cls, callable); if (ctx == NULL) { return NULL; } diff --git a/Modules/_sqlite/connection.h b/Modules/_sqlite/connection.h index c4cec857ddbfe..6a2aa1c8e080e 100644 --- a/Modules/_sqlite/connection.h +++ b/Modules/_sqlite/connection.h @@ -35,6 +35,7 @@ typedef struct _callback_context { PyObject *callable; + PyObject *module; pysqlite_state *state; } callback_context; From webhook-mailer at python.org Tue Oct 19 12:25:40 2021 From: webhook-mailer at python.org (markshannon) Date: Tue, 19 Oct 2021 16:25:40 -0000 Subject: [Python-checkins] Record cache hits for BINARY_SUBSCR specializations (GH-29060) Message-ID: https://github.com/python/cpython/commit/6e35b096ac07288a2e23909bb39f3a18c0c83951 commit: 6e35b096ac07288a2e23909bb39f3a18c0c83951 branch: main author: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> committer: markshannon date: 2021-10-19T17:25:31+01:00 summary: Record cache hits for BINARY_SUBSCR specializations (GH-29060) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 05bdcc5f7fab9..2fb60a84a112d 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -2242,6 +2242,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr Py_ssize_t index = ((PyLongObject*)sub)->ob_digit[0]; DEOPT_IF(index >= PyList_GET_SIZE(list), BINARY_SUBSCR); + record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *res = PyList_GET_ITEM(list, index); assert(res != NULL); @@ -2266,6 +2267,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr Py_ssize_t index = ((PyLongObject*)sub)->ob_digit[0]; DEOPT_IF(index >= PyTuple_GET_SIZE(tuple), BINARY_SUBSCR); + record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *res = PyTuple_GET_ITEM(tuple, index); assert(res != NULL); @@ -2280,6 +2282,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(BINARY_SUBSCR_DICT) { PyObject *dict = SECOND(); DEOPT_IF(!PyDict_CheckExact(SECOND()), BINARY_SUBSCR); + record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *sub = TOP(); PyObject *res = PyDict_GetItemWithError(dict, sub); From webhook-mailer at python.org Tue Oct 19 14:09:46 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:09:46 -0000 Subject: [Python-checkins] Add zoneinfo to the datetime documentation (GH-29038) Message-ID: https://github.com/python/cpython/commit/8e40ca127fa92d6113617c80710e0a077977a84d commit: 8e40ca127fa92d6113617c80710e0a077977a84d branch: main author: Paul Ganssle <1377457+pganssle at users.noreply.github.com> committer: ambv date: 2021-10-19T20:09:41+02:00 summary: Add zoneinfo to the datetime documentation (GH-29038) We should have done this way back when 3.9 was released, but it fell off the radar. Co-authored-by: Paul Ganssle files: A Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 196aa84473f13..0f8b70cdedb45 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -27,6 +27,9 @@ on efficient attribute extraction for output formatting and manipulation. Module :mod:`time` Time access and conversions. + Module :mod:`zoneinfo` + Concrete time zones representing the IANA time zone database. + Package `dateutil `_ Third-party library with expanded time zone and parsing support. @@ -2174,14 +2177,13 @@ only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). .. seealso:: - `dateutil.tz `_ + :mod:`zoneinfo` The :mod:`datetime` module has a basic :class:`timezone` class (for handling arbitrary fixed offsets from UTC) and its :attr:`timezone.utc` attribute (a UTC timezone instance). - *dateutil.tz* library brings the *IANA timezone database* - (also known as the Olson database) to Python, and its usage is - recommended. + ``zoneinfo`` brings the *IANA timezone database* (also known as the Olson + database) to Python, and its usage is recommended. `IANA timezone database `_ The Time Zone Database (often called tz, tzdata or zoneinfo) contains code diff --git a/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst new file mode 100644 index 0000000000000..382733ff91883 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst @@ -0,0 +1,3 @@ +Add references to :mod:`zoneinfo` in the :mod:`datetime` documentation, +mostly replacing outdated references to ``dateutil.tz``. Change by Paul +Ganssle. From webhook-mailer at python.org Tue Oct 19 14:18:59 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:18:59 -0000 Subject: [Python-checkins] bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) Message-ID: https://github.com/python/cpython/commit/7bafa0cf586227987d3d662264d491e3780024b7 commit: 7bafa0cf586227987d3d662264d491e3780024b7 branch: main author: Filipe La?ns committer: ambv date: 2021-10-19T20:18:50+02:00 summary: bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst M Doc/library/collections.abc.rst diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 07df1873bba58..132b0ce7192ac 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -104,6 +104,9 @@ example, knowing that a class supplies ``__getitem__``, ``__len__``, and ``__iter__`` is insufficient for distinguishing a :class:`Sequence` from a :class:`Mapping`. +.. versionadded:: 3.9 + These abstract classes now support ``[]``. See :ref:`types-genericalias` + and :pep:`585`. .. _collections-abstract-base-classes: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst new file mode 100644 index 0000000000000..fb817757a1487 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst @@ -0,0 +1 @@ +Add note about :pep:`585` in :mod:`collections.abc`. From webhook-mailer at python.org Tue Oct 19 14:28:35 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:28:35 -0000 Subject: [Python-checkins] bpo-40360: Make the 2to3 deprecation more obvious. (GH-29064) Message-ID: https://github.com/python/cpython/commit/fdbdf3f7359832820a11ece4c4b01581004d6fe7 commit: fdbdf3f7359832820a11ece4c4b01581004d6fe7 branch: main author: Gregory P. Smith committer: ambv date: 2021-10-19T20:28:27+02:00 summary: bpo-40360: Make the 2to3 deprecation more obvious. (GH-29064) files: M Doc/library/2to3.rst M Doc/whatsnew/3.11.rst diff --git a/Doc/library/2to3.rst b/Doc/library/2to3.rst index 2a13776e29336..9a1644a97c0ae 100644 --- a/Doc/library/2to3.rst +++ b/Doc/library/2to3.rst @@ -11,6 +11,11 @@ contains a rich set of fixers that will handle almost all code. 2to3 supporting library :mod:`lib2to3` is, however, a flexible and generic library, so it is possible to write your own fixers for 2to3. +.. deprecated-removed:: 3.11 3.13 + The ``lib2to3`` module was marked pending for deprecation in Python 3.9 + (raising :exc:`PendingDeprecationWarning` on import) and fully deprecated + in Python 3.11 (raising :exc:`DeprecationWarning`). The ``2to3`` tool is + part of that. It will be removed in Python 3.13. .. _2to3-using: diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 2e57f0cea53db..b583c2f31c4cc 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -308,9 +308,9 @@ CPython bytecode changes Deprecated ========== -* The :mod:`lib2to3` package is now deprecated and may not be able to parse - Python 3.10 or newer. See the :pep:`617` (New PEG parser for CPython). - (Contributed by Victor Stinner in :issue:`40360`.) +* The :mod:`lib2to3` package and ``2to3`` tool are now deprecated and may not + be able to parse Python 3.10 or newer. See the :pep:`617` (New PEG parser for + CPython). (Contributed by Victor Stinner in :issue:`40360`.) * :class:`webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. It is untested and undocumented and also not used by webbrowser itself. From webhook-mailer at python.org Tue Oct 19 14:35:39 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:35:39 -0000 Subject: [Python-checkins] bpo-45515: Add zoneinfo to the datetime documentation (GH-29038) (GH-29065) Message-ID: https://github.com/python/cpython/commit/8cef526a871695f8df4fac167139bbf3292396db commit: 8cef526a871695f8df4fac167139bbf3292396db branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-19T20:35:34+02:00 summary: bpo-45515: Add zoneinfo to the datetime documentation (GH-29038) (GH-29065) We should have done this way back when 3.9 was released, but it fell off the radar. Co-authored-by: Paul Ganssle (cherry picked from commit 8e40ca127fa92d6113617c80710e0a077977a84d) files: A Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 196aa84473f13..0f8b70cdedb45 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -27,6 +27,9 @@ on efficient attribute extraction for output formatting and manipulation. Module :mod:`time` Time access and conversions. + Module :mod:`zoneinfo` + Concrete time zones representing the IANA time zone database. + Package `dateutil `_ Third-party library with expanded time zone and parsing support. @@ -2174,14 +2177,13 @@ only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). .. seealso:: - `dateutil.tz `_ + :mod:`zoneinfo` The :mod:`datetime` module has a basic :class:`timezone` class (for handling arbitrary fixed offsets from UTC) and its :attr:`timezone.utc` attribute (a UTC timezone instance). - *dateutil.tz* library brings the *IANA timezone database* - (also known as the Olson database) to Python, and its usage is - recommended. + ``zoneinfo`` brings the *IANA timezone database* (also known as the Olson + database) to Python, and its usage is recommended. `IANA timezone database `_ The Time Zone Database (often called tz, tzdata or zoneinfo) contains code diff --git a/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst new file mode 100644 index 0000000000000..382733ff91883 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst @@ -0,0 +1,3 @@ +Add references to :mod:`zoneinfo` in the :mod:`datetime` documentation, +mostly replacing outdated references to ``dateutil.tz``. Change by Paul +Ganssle. From webhook-mailer at python.org Tue Oct 19 14:36:00 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:36:00 -0000 Subject: [Python-checkins] bpo-45515: Add zoneinfo to the datetime documentation (GH-29038) (GH-29066) Message-ID: https://github.com/python/cpython/commit/76578884c5c727c013cba9d14ced9040b5a0c8a1 commit: 76578884c5c727c013cba9d14ced9040b5a0c8a1 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-19T20:35:56+02:00 summary: bpo-45515: Add zoneinfo to the datetime documentation (GH-29038) (GH-29066) We should have done this way back when 3.9 was released, but it fell off the radar. Co-authored-by: Paul Ganssle (cherry picked from commit 8e40ca127fa92d6113617c80710e0a077977a84d) files: A Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst M Doc/library/datetime.rst diff --git a/Doc/library/datetime.rst b/Doc/library/datetime.rst index 196aa84473f13..0f8b70cdedb45 100644 --- a/Doc/library/datetime.rst +++ b/Doc/library/datetime.rst @@ -27,6 +27,9 @@ on efficient attribute extraction for output formatting and manipulation. Module :mod:`time` Time access and conversions. + Module :mod:`zoneinfo` + Concrete time zones representing the IANA time zone database. + Package `dateutil `_ Third-party library with expanded time zone and parsing support. @@ -2174,14 +2177,13 @@ only EST (fixed offset -5 hours), or only EDT (fixed offset -4 hours)). .. seealso:: - `dateutil.tz `_ + :mod:`zoneinfo` The :mod:`datetime` module has a basic :class:`timezone` class (for handling arbitrary fixed offsets from UTC) and its :attr:`timezone.utc` attribute (a UTC timezone instance). - *dateutil.tz* library brings the *IANA timezone database* - (also known as the Olson database) to Python, and its usage is - recommended. + ``zoneinfo`` brings the *IANA timezone database* (also known as the Olson + database) to Python, and its usage is recommended. `IANA timezone database `_ The Time Zone Database (often called tz, tzdata or zoneinfo) contains code diff --git a/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst new file mode 100644 index 0000000000000..382733ff91883 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-14-52-48.bpo-45515.aXdvm_.rst @@ -0,0 +1,3 @@ +Add references to :mod:`zoneinfo` in the :mod:`datetime` documentation, +mostly replacing outdated references to ``dateutil.tz``. Change by Paul +Ganssle. From webhook-mailer at python.org Tue Oct 19 14:40:38 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:40:38 -0000 Subject: [Python-checkins] bpo-42222: Improve tests for invalid argument types in randrange() (GH-29021) Message-ID: https://github.com/python/cpython/commit/574241632bd19e56ed488ee4d8999aefc6a8d7cd commit: 574241632bd19e56ed488ee4d8999aefc6a8d7cd branch: main author: Serhiy Storchaka committer: ambv date: 2021-10-19T20:40:30+02:00 summary: bpo-42222: Improve tests for invalid argument types in randrange() (GH-29021) files: M Lib/test/test_random.py diff --git a/Lib/test/test_random.py b/Lib/test/test_random.py index 3c5511d8c7f1a..cdae8897c2ae2 100644 --- a/Lib/test/test_random.py +++ b/Lib/test/test_random.py @@ -491,8 +491,10 @@ def test_randrange_errors(self): # Zero step raises_value_error(0, 42, 0) + raises_type_error(0, 42, 0.0) + raises_type_error(0, 0, 0.0) - # Non-integer start/stop/step + # Non-integer stop raises_type_error(3.14159) raises_type_error(3.0) raises_type_error(Fraction(3, 1)) @@ -501,10 +503,16 @@ def test_randrange_errors(self): raises_type_error(0, 2.0) raises_type_error(0, Fraction(2, 1)) raises_type_error(0, '2') + raises_type_error(0, 2.71827, 2) + + # Non-integer start + raises_type_error(2.71827, 5) + raises_type_error(2.0, 5) + raises_type_error(Fraction(2, 1), 5) + raises_type_error('2', 5) + raises_type_error(2.71827, 5, 2) # Non-integer step - raises_type_error(0, 42, 1.0) - raises_type_error(0, 0, 1.0) raises_type_error(0, 42, 3.14159) raises_type_error(0, 42, 3.0) raises_type_error(0, 42, Fraction(3, 1)) @@ -512,23 +520,6 @@ def test_randrange_errors(self): raises_type_error(0, 42, 1.0) raises_type_error(0, 0, 1.0) - def test_randrange_argument_handling(self): - randrange = self.gen.randrange - with self.assertRaises(TypeError): - randrange(10.0, 20, 2) - with self.assertRaises(TypeError): - randrange(10, 20.0, 2) - with self.assertRaises(TypeError): - randrange(10, 20, 1.0) - with self.assertRaises(TypeError): - randrange(10, 20, 2.0) - with self.assertRaises(TypeError): - randrange(10.5) - with self.assertRaises(TypeError): - randrange(10, 20.5) - with self.assertRaises(TypeError): - randrange(10, 20, 1.5) - def test_randrange_step(self): # bpo-42772: When stop is None, the step argument was being ignored. randrange = self.gen.randrange From webhook-mailer at python.org Tue Oct 19 14:41:38 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:41:38 -0000 Subject: [Python-checkins] bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) (GH-29067) Message-ID: https://github.com/python/cpython/commit/092ec4b9d144c493cb514df0978638fdd2d9622a commit: 092ec4b9d144c493cb514df0978638fdd2d9622a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-19T20:41:33+02:00 summary: bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) (GH-29067) Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa (cherry picked from commit 7bafa0cf586227987d3d662264d491e3780024b7) files: A Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst M Doc/library/collections.abc.rst diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 0abc87f919de6..2c941b47748d3 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -104,6 +104,9 @@ example, knowing that a class supplies ``__getitem__``, ``__len__``, and ``__iter__`` is insufficient for distinguishing a :class:`Sequence` from a :class:`Mapping`. +.. versionadded:: 3.9 + These abstract classes now support ``[]``. See :ref:`types-genericalias` + and :pep:`585`. .. _collections-abstract-base-classes: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst new file mode 100644 index 0000000000000..fb817757a1487 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst @@ -0,0 +1 @@ +Add note about :pep:`585` in :mod:`collections.abc`. From webhook-mailer at python.org Tue Oct 19 14:42:18 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:42:18 -0000 Subject: [Python-checkins] bpo-42174: fallback to sane values if the columns or lines are 0 in get_terminal_size (GH-29046) Message-ID: https://github.com/python/cpython/commit/236e301b8ad9b78ee880baf12e98a826113dc59b commit: 236e301b8ad9b78ee880baf12e98a826113dc59b branch: main author: Filipe La?ns committer: ambv date: 2021-10-19T20:42:13+02:00 summary: bpo-42174: fallback to sane values if the columns or lines are 0 in get_terminal_size (GH-29046) I considered only falling back when both were 0, but that still seems wrong, and the highly popular rich[1] library does it this way, so I thought we should probably inherit that behavior. [1] https://github.com/willmcgugan/rich Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-10-19-01-30-57.bpo-42174.O2w9bi.rst M Doc/library/shutil.rst M Lib/shutil.py diff --git a/Doc/library/shutil.rst b/Doc/library/shutil.rst index 11c6707492167..22d6dba9e1a9c 100644 --- a/Doc/library/shutil.rst +++ b/Doc/library/shutil.rst @@ -804,6 +804,10 @@ Querying the size of the output terminal .. versionadded:: 3.3 + .. versionchanged:: 3.11 + The ``fallback`` values are also used if :func:`os.get_terminal_size` + returns zeroes. + .. _`fcopyfile`: http://www.manpagez.com/man/3/copyfile/ diff --git a/Lib/shutil.py b/Lib/shutil.py index e544498fca7c8..949e024853c1d 100644 --- a/Lib/shutil.py +++ b/Lib/shutil.py @@ -1372,9 +1372,9 @@ def get_terminal_size(fallback=(80, 24)): # os.get_terminal_size() is unsupported size = os.terminal_size(fallback) if columns <= 0: - columns = size.columns + columns = size.columns or fallback[0] if lines <= 0: - lines = size.lines + lines = size.lines or fallback[1] return os.terminal_size((columns, lines)) diff --git a/Misc/NEWS.d/next/Library/2021-10-19-01-30-57.bpo-42174.O2w9bi.rst b/Misc/NEWS.d/next/Library/2021-10-19-01-30-57.bpo-42174.O2w9bi.rst new file mode 100644 index 0000000000000..412582ded3e12 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-19-01-30-57.bpo-42174.O2w9bi.rst @@ -0,0 +1,2 @@ +:meth:`shutil.get_terminal_size` now falls back to sane values if the column +or line count are 0. From webhook-mailer at python.org Tue Oct 19 14:50:21 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 18:50:21 -0000 Subject: [Python-checkins] [3.9] bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) (GH-29068) Message-ID: https://github.com/python/cpython/commit/a18e4e9c15a0be833e01c3892206661fc91e6918 commit: a18e4e9c15a0be833e01c3892206661fc91e6918 branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-19T20:50:09+02:00 summary: [3.9] bpo-45449: add note about PEP 585 in collections.abc's documentation (GH-29047) (GH-29068) Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa . (cherry picked from commit 7bafa0cf586227987d3d662264d491e3780024b7) Co-authored-by: Filipe La?ns files: A Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst M Doc/library/collections.abc.rst diff --git a/Doc/library/collections.abc.rst b/Doc/library/collections.abc.rst index 2345e78a17e4f..cb4f767097169 100644 --- a/Doc/library/collections.abc.rst +++ b/Doc/library/collections.abc.rst @@ -24,6 +24,9 @@ This module provides :term:`abstract base classes ` that can be used to test whether a class provides a particular interface; for example, whether it is hashable or whether it is a mapping. +.. versionadded:: 3.9 + These abstract classes now support ``[]``. See :ref:`types-genericalias` + and :pep:`585`. .. _collections-abstract-base-classes: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst new file mode 100644 index 0000000000000..fb817757a1487 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-19-01-41-40.bpo-45449.fjHZJc.rst @@ -0,0 +1 @@ +Add note about :pep:`585` in :mod:`collections.abc`. From webhook-mailer at python.org Tue Oct 19 15:11:37 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 19:11:37 -0000 Subject: [Python-checkins] bpo-45500: Rewrite test_dbm (GH-29002) Message-ID: https://github.com/python/cpython/commit/975b94b9de969777218e96a9950c1dab2dab65a0 commit: 975b94b9de969777218e96a9950c1dab2dab65a0 branch: main author: Serhiy Storchaka committer: ambv date: 2021-10-19T21:11:28+02:00 summary: bpo-45500: Rewrite test_dbm (GH-29002) * Generate test classes at import time. It allows to filter them when run with unittest. E.g: "./python -m unittest test.test_dbm.TestCase_gnu -v". * Create a database class in a new directory which will be removed after test. It guarantees that all created files and directories be removed and will not conflict with other dbm tests. * Restore dbm._defaultmod after tests. Previously it was set to the last dbm module (dbm.dumb) which affected other tests. * Enable the whichdb test for dbm.dumb. * Move test_keys to the correct test class. It does not test whichdb(). * Remove some outdated code and comments. files: M Lib/test/test_dbm.py diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py index cf0758653ff38..f21eebc6530c7 100644 --- a/Lib/test/test_dbm.py +++ b/Lib/test/test_dbm.py @@ -1,24 +1,21 @@ """Test script for the dbm.open function based on testdumbdbm.py""" import unittest -import glob +import dbm import os from test.support import import_helper from test.support import os_helper -# Skip tests if dbm module doesn't exist. -dbm = import_helper.import_module('dbm') - try: from dbm import ndbm except ImportError: ndbm = None -_fname = os_helper.TESTFN +dirname = os_helper.TESTFN +_fname = os.path.join(dirname, os_helper.TESTFN) # -# Iterates over every database module supported by dbm currently available, -# setting dbm to use each in turn, and yielding that module +# Iterates over every database module supported by dbm currently available. # def dbm_iterator(): for name in dbm._names: @@ -32,11 +29,12 @@ def dbm_iterator(): # # Clean up all scratch databases we might have created during testing # -def delete_files(): - # we don't know the precise name the underlying database uses - # so we use glob to locate all names - for f in glob.glob(glob.escape(_fname) + "*"): - os_helper.unlink(f) +def cleaunup_test_dir(): + os_helper.rmtree(dirname) + +def setup_test_dir(): + cleaunup_test_dir() + os.mkdir(dirname) class AnyDBMTestCase: @@ -144,86 +142,76 @@ def read_helper(self, f): for key in self._dict: self.assertEqual(self._dict[key], f[key.encode("ascii")]) - def tearDown(self): - delete_files() + def test_keys(self): + with dbm.open(_fname, 'c') as d: + self.assertEqual(d.keys(), []) + a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] + for k, v in a: + d[k] = v + self.assertEqual(sorted(d.keys()), sorted(k for (k, v) in a)) + for k, v in a: + self.assertIn(k, d) + self.assertEqual(d[k], v) + self.assertNotIn(b'xxx', d) + self.assertRaises(KeyError, lambda: d[b'xxx']) def setUp(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) dbm._defaultmod = self.module - delete_files() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() class WhichDBTestCase(unittest.TestCase): def test_whichdb(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) _bytes_fname = os.fsencode(_fname) - for path in [_fname, os_helper.FakePath(_fname), - _bytes_fname, os_helper.FakePath(_bytes_fname)]: - for module in dbm_iterator(): - # Check whether whichdb correctly guesses module name - # for databases opened with "module" module. - # Try with empty files first - name = module.__name__ - if name == 'dbm.dumb': - continue # whichdb can't support dbm.dumb - delete_files() - f = module.open(path, 'c') - f.close() + fnames = [_fname, os_helper.FakePath(_fname), + _bytes_fname, os_helper.FakePath(_bytes_fname)] + for module in dbm_iterator(): + # Check whether whichdb correctly guesses module name + # for databases opened with "module" module. + name = module.__name__ + setup_test_dir() + dbm._defaultmod = module + # Try with empty files first + with module.open(_fname, 'c'): pass + for path in fnames: self.assertEqual(name, self.dbm.whichdb(path)) - # Now add a key - f = module.open(path, 'w') + # Now add a key + with module.open(_fname, 'w') as f: f[b"1"] = b"1" # and test that we can find it self.assertIn(b"1", f) # and read it self.assertEqual(f[b"1"], b"1") - f.close() + for path in fnames: self.assertEqual(name, self.dbm.whichdb(path)) @unittest.skipUnless(ndbm, reason='Test requires ndbm') def test_whichdb_ndbm(self): # Issue 17198: check that ndbm which is referenced in whichdb is defined - db_file = '{}_ndbm.db'.format(_fname) - with open(db_file, 'w'): - self.addCleanup(os_helper.unlink, db_file) - db_file_bytes = os.fsencode(db_file) - self.assertIsNone(self.dbm.whichdb(db_file[:-3])) - self.assertIsNone(self.dbm.whichdb(os_helper.FakePath(db_file[:-3]))) - self.assertIsNone(self.dbm.whichdb(db_file_bytes[:-3])) - self.assertIsNone(self.dbm.whichdb(os_helper.FakePath(db_file_bytes[:-3]))) - - def tearDown(self): - delete_files() + with open(_fname + '.db', 'wb'): pass + _bytes_fname = os.fsencode(_fname) + fnames = [_fname, os_helper.FakePath(_fname), + _bytes_fname, os_helper.FakePath(_bytes_fname)] + for path in fnames: + self.assertIsNone(self.dbm.whichdb(path)) def setUp(self): - delete_files() - self.filename = os_helper.TESTFN - self.d = dbm.open(self.filename, 'c') - self.d.close() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() self.dbm = import_helper.import_fresh_module('dbm') - def test_keys(self): - self.d = dbm.open(self.filename, 'c') - self.assertEqual(self.d.keys(), []) - a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] - for k, v in a: - self.d[k] = v - self.assertEqual(sorted(self.d.keys()), sorted(k for (k, v) in a)) - for k, v in a: - self.assertIn(k, self.d) - self.assertEqual(self.d[k], v) - self.assertNotIn(b'xxx', self.d) - self.assertRaises(KeyError, lambda: self.d[b'xxx']) - self.d.close() - - -def load_tests(loader, tests, pattern): - classes = [] - for mod in dbm_iterator(): - classes.append(type("TestCase-" + mod.__name__, - (AnyDBMTestCase, unittest.TestCase), - {'module': mod})) - for c in classes: - tests.addTest(loader.loadTestsFromTestCase(c)) - return tests + +for mod in dbm_iterator(): + assert mod.__name__.startswith('dbm.') + suffix = mod.__name__[4:] + testname = f'TestCase_{suffix}' + globals()[testname] = type(testname, + (AnyDBMTestCase, unittest.TestCase), + {'module': mod}) + if __name__ == "__main__": unittest.main() From webhook-mailer at python.org Tue Oct 19 15:13:32 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 19:13:32 -0000 Subject: [Python-checkins] [doc]: Fix missing space in c-api/init.rst and add rstlint rule (GH-28988) Message-ID: https://github.com/python/cpython/commit/bda69abe849b37467350d3750ae24d356230c940 commit: bda69abe849b37467350d3750ae24d356230c940 branch: main author: Julien Palard committer: ambv date: 2021-10-19T21:13:24+02:00 summary: [doc]: Fix missing space in c-api/init.rst and add rstlint rule (GH-28988) files: M Doc/c-api/init.rst M Doc/tools/rstlint.py diff --git a/Doc/c-api/init.rst b/Doc/c-api/init.rst index 5e817268a2036..09dfc68fee57d 100644 --- a/Doc/c-api/init.rst +++ b/Doc/c-api/init.rst @@ -1177,7 +1177,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`. Suspend tracing and profiling in the Python thread state *tstate*. - Resume them using the:c:func:`PyThreadState_LeaveTracing` function. + Resume them using the :c:func:`PyThreadState_LeaveTracing` function. .. versionadded:: 3.11 @@ -1185,7 +1185,7 @@ All of the following functions must be called after :c:func:`Py_Initialize`. .. c:function:: void PyThreadState_LeaveTracing(PyThreadState *tstate) Resume tracing and profiling in the Python thread state *tstate* suspended - by the:c:func:`PyThreadState_EnterTracing` function. + by the :c:func:`PyThreadState_EnterTracing` function. See also :c:func:`PyEval_SetTrace` and :c:func:`PyEval_SetProfile` functions. diff --git a/Doc/tools/rstlint.py b/Doc/tools/rstlint.py index 045b7d7c945df..33cbaadfce944 100755 --- a/Doc/tools/rstlint.py +++ b/Doc/tools/rstlint.py @@ -43,10 +43,10 @@ ] roles = [ - ":class:", - ":func:", - ":meth:", - ":mod:", + "(? https://github.com/python/cpython/commit/1d8cb01e23ab5fc626d4deda216f9a9a19a644a2 commit: 1d8cb01e23ab5fc626d4deda216f9a9a19a644a2 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-19T21:14:36+02:00 summary: bpo-45195: Fix test_readline.test_nonascii() (GH-28329) (GH-28984) Fix test_readline.test_nonascii(): sometimes, the newline character is not written at the end, so don't expect it in the output. (cherry picked from commit 797c8eb9ef511f0c25f10a453b35c4d2fe383c30) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst M Lib/test/test_readline.py diff --git a/Lib/test/test_readline.py b/Lib/test/test_readline.py index 67ee9b7f7cfb1..3574657cf3635 100644 --- a/Lib/test/test_readline.py +++ b/Lib/test/test_readline.py @@ -232,7 +232,9 @@ def display(substitution, matches, longest_match_length): self.assertIn(b"matches ['t\\xebnt', 't\\xebxt']\r\n", output) expected = br"'[\xefnserted]|t\xebxt[after]'" self.assertIn(b"result " + expected + b"\r\n", output) - self.assertIn(b"history " + expected + b"\r\n", output) + # bpo-45195: Sometimes, the newline character is not written at the + # end, so don't expect it in the output. + self.assertIn(b"history " + expected, output) # We have 2 reasons to skip this test: # - readline: history size was added in 6.0 diff --git a/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst b/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst new file mode 100644 index 0000000000000..16a1f4440483c --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-09-14-13-16-18.bpo-45195.EyQR1G.rst @@ -0,0 +1,3 @@ +Fix test_readline.test_nonascii(): sometimes, the newline character is not +written at the end, so don't expect it in the output. Patch by Victor +Stinner. From webhook-mailer at python.org Tue Oct 19 15:15:11 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 19:15:11 -0000 Subject: [Python-checkins] bpo-45310: Fix parrallel shared memory tests (GH-28661) (GH-28979) Message-ID: https://github.com/python/cpython/commit/7f70ba36706219181546c05363097f732827a081 commit: 7f70ba36706219181546c05363097f732827a081 branch: 3.8 author: Victor Stinner committer: ambv date: 2021-10-19T21:15:06+02:00 summary: bpo-45310: Fix parrallel shared memory tests (GH-28661) (GH-28979) Add a PID to names of POSIX shared memory objects to allow running multiprocessing tests (test_multiprocessing_fork, test_multiprocessing_spawn, etc) in parallel. (cherry picked from commit eb4495e8e275c83d691add116c4f2b74e73e3cc8) Co-authored-by: Serhiy Storchaka files: M Lib/test/_test_multiprocessing.py diff --git a/Lib/test/_test_multiprocessing.py b/Lib/test/_test_multiprocessing.py index ee9b47bb5f153..643accc82c49d 100644 --- a/Lib/test/_test_multiprocessing.py +++ b/Lib/test/_test_multiprocessing.py @@ -3749,12 +3749,19 @@ def _attach_existing_shmem_then_write(shmem_name_or_obj, binary_data): local_sms.buf[:len(binary_data)] = binary_data local_sms.close() + def _new_shm_name(self, prefix): + # Add a PID to the name of a POSIX shared memory object to allow + # running multiprocessing tests (test_multiprocessing_fork, + # test_multiprocessing_spawn, etc) in parallel. + return prefix + str(os.getpid()) + def test_shared_memory_basics(self): - sms = shared_memory.SharedMemory('test01_tsmb', create=True, size=512) + name_tsmb = self._new_shm_name('test01_tsmb') + sms = shared_memory.SharedMemory(name_tsmb, create=True, size=512) self.addCleanup(sms.unlink) # Verify attributes are readable. - self.assertEqual(sms.name, 'test01_tsmb') + self.assertEqual(sms.name, name_tsmb) self.assertGreaterEqual(sms.size, 512) self.assertGreaterEqual(len(sms.buf), sms.size) @@ -3763,12 +3770,12 @@ def test_shared_memory_basics(self): self.assertEqual(sms.buf[0], 42) # Attach to existing shared memory segment. - also_sms = shared_memory.SharedMemory('test01_tsmb') + also_sms = shared_memory.SharedMemory(name_tsmb) self.assertEqual(also_sms.buf[0], 42) also_sms.close() # Attach to existing shared memory segment but specify a new size. - same_sms = shared_memory.SharedMemory('test01_tsmb', size=20*sms.size) + same_sms = shared_memory.SharedMemory(name_tsmb, size=20*sms.size) self.assertLess(same_sms.size, 20*sms.size) # Size was ignored. same_sms.close() @@ -3779,17 +3786,17 @@ def test_shared_memory_basics(self): # manages unlinking on its own and unlink() does nothing). # True release of shared memory segment does not necessarily # happen until process exits, depending on the OS platform. + name_dblunlink = self._new_shm_name('test01_dblunlink') + sms_uno = shared_memory.SharedMemory( + name_dblunlink, + create=True, + size=5000 + ) with self.assertRaises(FileNotFoundError): - sms_uno = shared_memory.SharedMemory( - 'test01_dblunlink', - create=True, - size=5000 - ) - try: self.assertGreaterEqual(sms_uno.size, 5000) - sms_duo = shared_memory.SharedMemory('test01_dblunlink') + sms_duo = shared_memory.SharedMemory(name_dblunlink) sms_duo.unlink() # First shm_unlink() call. sms_duo.close() sms_uno.close() @@ -3801,7 +3808,7 @@ def test_shared_memory_basics(self): # Attempting to create a new shared memory segment with a # name that is already in use triggers an exception. there_can_only_be_one_sms = shared_memory.SharedMemory( - 'test01_tsmb', + name_tsmb, create=True, size=512 ) @@ -3815,7 +3822,7 @@ def test_shared_memory_basics(self): # case of MacOS/darwin, requesting a smaller size is disallowed. class OptionalAttachSharedMemory(shared_memory.SharedMemory): _flags = os.O_CREAT | os.O_RDWR - ok_if_exists_sms = OptionalAttachSharedMemory('test01_tsmb') + ok_if_exists_sms = OptionalAttachSharedMemory(name_tsmb) self.assertEqual(ok_if_exists_sms.size, sms.size) ok_if_exists_sms.close() @@ -4003,10 +4010,11 @@ def test_shared_memory_ShareableList_basics(self): self.assertEqual(sl.count(b'adios'), 0) # Exercise creating a duplicate. - sl_copy = shared_memory.ShareableList(sl, name='test03_duplicate') + name_duplicate = self._new_shm_name('test03_duplicate') + sl_copy = shared_memory.ShareableList(sl, name=name_duplicate) try: self.assertNotEqual(sl.shm.name, sl_copy.shm.name) - self.assertEqual('test03_duplicate', sl_copy.shm.name) + self.assertEqual(name_duplicate, sl_copy.shm.name) self.assertEqual(list(sl), list(sl_copy)) self.assertEqual(sl.format, sl_copy.format) sl_copy[-1] = 77 From webhook-mailer at python.org Tue Oct 19 15:15:34 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 19:15:34 -0000 Subject: [Python-checkins] bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) (GH-28978) Message-ID: https://github.com/python/cpython/commit/67e10be3fee7c69d4ece26e75a0b7a84dab68ce5 commit: 67e10be3fee7c69d4ece26e75a0b7a84dab68ce5 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-19T21:15:29+02:00 summary: bpo-44849: Fix os.set_inheritable() on FreeBSD 14 with O_PATH (GH-27623) (GH-28978) Fix the os.set_inheritable() function on FreeBSD 14 for file descriptor opened with the O_PATH flag: ignore the EBADF error on ioctl(), fallback on the fcntl() implementation. (cherry picked from commit c24896c0e3b32c8a9f614ef51366007b67d5c665) Co-authored-by: Victor Stinner files: A Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst M Python/fileutils.c diff --git a/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst b/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst new file mode 100644 index 0000000000000..b1f225485ddef --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-06-13-00-28.bpo-44849.O78F_f.rst @@ -0,0 +1,4 @@ +Fix the :func:`os.set_inheritable` function on FreeBSD 14 for file descriptor +opened with the :data:`~os.O_PATH` flag: ignore the :data:`~errno.EBADF` +error on ``ioctl()``, fallback on the ``fcntl()`` implementation. Patch by +Victor Stinner. diff --git a/Python/fileutils.c b/Python/fileutils.c index 3e1311c978686..f0bdd9c17371d 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -1205,10 +1205,11 @@ set_inheritable(int fd, int inheritable, int raise, int *atomic_flag_works) return 0; } -#ifdef __linux__ +#ifdef O_PATH if (errno == EBADF) { - // On Linux, ioctl(FIOCLEX) will fail with EBADF for O_PATH file descriptors - // Fall through to the fcntl() path + // bpo-44849: On Linux and FreeBSD, ioctl(FIOCLEX) fails with EBADF + // on O_PATH file descriptors. Fall through to the fcntl() + // implementation. } else #endif From webhook-mailer at python.org Tue Oct 19 15:24:21 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 19:24:21 -0000 Subject: [Python-checkins] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) Message-ID: https://github.com/python/cpython/commit/a106343f632a99c8ebb0136fa140cf189b4a6a57 commit: a106343f632a99c8ebb0136fa140cf189b4a6a57 branch: main author: Pablo Galindo Salgado committer: ambv date: 2021-10-19T21:24:12+02:00 summary: bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing. files: A Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst M Lib/test/test_exceptions.py M Parser/parser.c M Parser/pegen.c M Tools/peg_generator/pegen/c_generator.py diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 02489a2bd9215..0f8a8f134b61f 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -209,6 +209,10 @@ def check(self, src, lineno, offset, encoding='utf-8'): src = src.decode(encoding, 'replace') line = src.split('\n')[lineno-1] self.assertIn(line, cm.exception.text) + + def test_error_offset_continuation_characters(self): + check = self.check + check('"\\\n"(1 for c in I,\\\n\\', 2, 2) def testSyntaxErrorOffset(self): check = self.check diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst new file mode 100644 index 0000000000000..97e29813ab266 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst @@ -0,0 +1,2 @@ +Fix parser crash when reporting errors involving invalid continuation +characters. Patch by Pablo Galindo. diff --git a/Parser/parser.c b/Parser/parser.c index beb2176e3bbf2..47722f3baf1e6 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -946,7 +946,7 @@ file_rule(Parser *p) void *a; Token * endmarker_var; if ( - (a = statements_rule(p), 1) // statements? + (a = statements_rule(p), !p->error_indicator) // statements? && (endmarker_var = _PyPegen_expect_token(p, ENDMARKER)) // token='ENDMARKER' ) @@ -1085,7 +1085,7 @@ func_type_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = type_expressions_rule(p), 1) // type_expressions? + (a = type_expressions_rule(p), !p->error_indicator) // type_expressions? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -1441,7 +1441,7 @@ simple_stmts_rule(Parser *p) if ( (a = (asdl_stmt_seq*)_gather_4_rule(p)) // ';'.simple_stmt+ && - (_opt_var = _PyPegen_expect_token(p, 13), 1) // ';'? + (_opt_var = _PyPegen_expect_token(p, 13), !p->error_indicator) // ';'? && (newline_var = _PyPegen_expect_token(p, NEWLINE)) // token='NEWLINE' ) @@ -2061,7 +2061,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_11_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_11_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ':' expression ['=' annotated_rhs]")); @@ -2103,7 +2103,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_13_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_13_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "('(' single_target ')' | single_subscript_attribute_target) ':' expression ['=' annotated_rhs]")); @@ -2144,7 +2144,7 @@ assignment_rule(Parser *p) && _PyPegen_lookahead_with_int(0, _PyPegen_expect_token, p, 22) // token='=' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "((star_targets '='))+ (yield_expr | star_expressions) !'=' TYPE_COMMENT?")); @@ -2668,7 +2668,7 @@ return_stmt_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 519)) // token='return' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ return_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'return' star_expressions?")); @@ -2733,7 +2733,7 @@ raise_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_17_rule(p), 1) // ['from' expression] + (b = _tmp_17_rule(p), !p->error_indicator) // ['from' expression] ) { D(fprintf(stderr, "%*c+ raise_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'raise' expression ['from' expression]")); @@ -3097,7 +3097,7 @@ assert_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_23_rule(p), 1) // [',' expression] + (b = _tmp_23_rule(p), !p->error_indicator) // [',' expression] ) { D(fprintf(stderr, "%*c+ assert_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'assert' expression [',' expression]")); @@ -3400,7 +3400,7 @@ import_from_targets_rule(Parser *p) && (a = import_from_as_names_rule(p)) // import_from_as_names && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -3569,7 +3569,7 @@ import_from_as_name_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_28_rule(p), 1) // ['as' NAME] + (b = _tmp_28_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ import_from_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ['as' NAME]")); @@ -3672,7 +3672,7 @@ dotted_as_name_rule(Parser *p) if ( (a = dotted_name_rule(p)) // dotted_name && - (b = _tmp_31_rule(p), 1) // ['as' NAME] + (b = _tmp_31_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ dotted_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "dotted_name ['as' NAME]")); @@ -4055,7 +4055,7 @@ class_def_raw_rule(Parser *p) && (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_33_rule(p), 1) // ['(' arguments? ')'] + (b = _tmp_33_rule(p), !p->error_indicator) // ['(' arguments? ')'] && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && @@ -4217,15 +4217,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_34_rule(p), 1) // ['->' expression] + (a = _tmp_34_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -4277,15 +4277,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_35_rule(p), 1) // ['->' expression] + (a = _tmp_35_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -4406,7 +4406,7 @@ parameters_rule(Parser *p) && (c = _loop0_37_rule(p)) // param_with_default* && - (d = star_etc_rule(p), 1) // star_etc? + (d = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_no_default param_no_default* param_with_default* star_etc?")); @@ -4436,7 +4436,7 @@ parameters_rule(Parser *p) && (b = _loop0_38_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_with_default param_with_default* star_etc?")); @@ -4466,7 +4466,7 @@ parameters_rule(Parser *p) && (b = _loop0_40_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_no_default+ param_with_default* star_etc?")); @@ -4493,7 +4493,7 @@ parameters_rule(Parser *p) if ( (a = _loop1_41_rule(p)) // param_with_default+ && - (b = star_etc_rule(p), 1) // star_etc? + (b = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_with_default+ star_etc?")); @@ -4731,7 +4731,7 @@ star_etc_rule(Parser *p) && (b = _loop0_48_rule(p)) // param_maybe_default* && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' param_no_default param_maybe_default* kwds?")); @@ -4764,7 +4764,7 @@ star_etc_rule(Parser *p) && (b = _loop1_49_rule(p)) // param_maybe_default+ && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' param_maybe_default+ kwds?")); @@ -4898,7 +4898,7 @@ param_no_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_no_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param ',' TYPE_COMMENT?")); @@ -4925,7 +4925,7 @@ param_no_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -4977,7 +4977,7 @@ param_with_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_with_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default ',' TYPE_COMMENT?")); @@ -5007,7 +5007,7 @@ param_with_default_rule(Parser *p) && (c = default_rule(p)) // default && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -5057,11 +5057,11 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_maybe_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default? ',' TYPE_COMMENT?")); @@ -5089,9 +5089,9 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -5146,7 +5146,7 @@ param_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = annotation_rule(p), 1) // annotation? + (b = annotation_rule(p), !p->error_indicator) // annotation? ) { D(fprintf(stderr, "%*c+ param[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME annotation?")); @@ -5372,7 +5372,7 @@ if_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ if_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'if' named_expression ':' block else_block?")); @@ -5510,7 +5510,7 @@ elif_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ elif_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'elif' named_expression ':' block else_block?")); @@ -5666,7 +5666,7 @@ while_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ while_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'while' named_expression ':' block else_block?")); @@ -5768,11 +5768,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'for' star_targets 'in' ~ star_expressions &&':' TYPE_COMMENT? block else_block?")); @@ -5832,11 +5832,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "ASYNC 'for' star_targets 'in' ~ star_expressions &&':' TYPE_COMMENT? block else_block?")); @@ -5956,7 +5956,7 @@ with_stmt_rule(Parser *p) && (a = (asdl_withitem_seq*)_gather_50_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -6005,7 +6005,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -6056,7 +6056,7 @@ with_stmt_rule(Parser *p) && (a = (asdl_withitem_seq*)_gather_54_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -6108,7 +6108,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -6360,9 +6360,9 @@ try_stmt_rule(Parser *p) && (ex = (asdl_excepthandler_seq*)_loop1_59_rule(p)) // except_block+ && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? && - (f = finally_block_rule(p), 1) // finally_block? + (f = finally_block_rule(p), !p->error_indicator) // finally_block? ) { D(fprintf(stderr, "%*c+ try_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'try' &&':' block except_block+ else_block? finally_block?")); @@ -6452,7 +6452,7 @@ except_block_rule(Parser *p) && (e = expression_rule(p)) // expression && - (t = _tmp_60_rule(p), 1) // ['as' NAME] + (t = _tmp_60_rule(p), !p->error_indicator) // ['as' NAME] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -6743,7 +6743,7 @@ subject_expr_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (values = star_named_expressions_rule(p), 1) // star_named_expressions? + (values = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? ) { D(fprintf(stderr, "%*c+ subject_expr[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_named_expression ',' star_named_expressions?")); @@ -6839,7 +6839,7 @@ case_block_rule(Parser *p) && (pattern = patterns_rule(p)) // patterns && - (guard = guard_rule(p), 1) // guard? + (guard = guard_rule(p), !p->error_indicator) // guard? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -8592,7 +8592,7 @@ sequence_pattern_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (patterns = maybe_sequence_pattern_rule(p), 1) // maybe_sequence_pattern? + (patterns = maybe_sequence_pattern_rule(p), !p->error_indicator) // maybe_sequence_pattern? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -8631,7 +8631,7 @@ sequence_pattern_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (patterns = open_sequence_pattern_rule(p), 1) // open_sequence_pattern? + (patterns = open_sequence_pattern_rule(p), !p->error_indicator) // open_sequence_pattern? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -8689,7 +8689,7 @@ open_sequence_pattern_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (patterns = maybe_sequence_pattern_rule(p), 1) // maybe_sequence_pattern? + (patterns = maybe_sequence_pattern_rule(p), !p->error_indicator) // maybe_sequence_pattern? ) { D(fprintf(stderr, "%*c+ open_sequence_pattern[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "maybe_star_pattern ',' maybe_sequence_pattern?")); @@ -8734,7 +8734,7 @@ maybe_sequence_pattern_rule(Parser *p) if ( (patterns = _gather_68_rule(p)) // ','.maybe_star_pattern+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ maybe_sequence_pattern[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.maybe_star_pattern+ ','?")); @@ -8985,7 +8985,7 @@ mapping_pattern_rule(Parser *p) && (rest = double_star_pattern_rule(p)) // double_star_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -9034,7 +9034,7 @@ mapping_pattern_rule(Parser *p) && (rest = double_star_pattern_rule(p)) // double_star_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_2 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -9077,7 +9077,7 @@ mapping_pattern_rule(Parser *p) && (items = items_pattern_rule(p)) // items_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -9320,7 +9320,7 @@ class_pattern_rule(Parser *p) && (patterns = positional_patterns_rule(p)) // positional_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -9366,7 +9366,7 @@ class_pattern_rule(Parser *p) && (keywords = keyword_patterns_rule(p)) // keyword_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -9418,7 +9418,7 @@ class_pattern_rule(Parser *p) && (keywords = keyword_patterns_rule(p)) // keyword_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_2 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -9629,7 +9629,7 @@ expressions_rule(Parser *p) && (b = _loop1_77_rule(p)) // ((',' expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ((',' expression))+ ','?")); @@ -9922,7 +9922,7 @@ yield_expr_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 570)) // token='yield' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ yield_expr[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'yield' star_expressions?")); @@ -9991,7 +9991,7 @@ star_expressions_rule(Parser *p) && (b = _loop1_78_rule(p)) // ((',' star_expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_expression ((',' star_expression))+ ','?")); @@ -10186,7 +10186,7 @@ star_named_expressions_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_79_rule(p)) // ','.star_named_expression+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_named_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_named_expression+ ','?")); @@ -12864,7 +12864,7 @@ primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -13016,7 +13016,7 @@ slices_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_85_rule(p)) // ','.slice+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ slices[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.slice+ ','?")); @@ -13078,13 +13078,13 @@ slice_rule(Parser *p) void *b; void *c; if ( - (a = expression_rule(p), 1) // expression? + (a = expression_rule(p), !p->error_indicator) // expression? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (b = expression_rule(p), 1) // expression? + (b = expression_rule(p), !p->error_indicator) // expression? && - (c = _tmp_87_rule(p), 1) // [':' expression?] + (c = _tmp_87_rule(p), !p->error_indicator) // [':' expression?] ) { D(fprintf(stderr, "%*c+ slice[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression? ':' expression? [':' expression?]")); @@ -13528,7 +13528,7 @@ lambdef_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 583)) // token='lambda' && - (a = lambda_params_rule(p), 1) // lambda_params? + (a = lambda_params_rule(p), !p->error_indicator) // lambda_params? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -13651,7 +13651,7 @@ lambda_parameters_rule(Parser *p) && (c = _loop0_93_rule(p)) // lambda_param_with_default* && - (d = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (d = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_no_default lambda_param_no_default* lambda_param_with_default* lambda_star_etc?")); @@ -13681,7 +13681,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_94_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_with_default lambda_param_with_default* lambda_star_etc?")); @@ -13711,7 +13711,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_96_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_no_default+ lambda_param_with_default* lambda_star_etc?")); @@ -13738,7 +13738,7 @@ lambda_parameters_rule(Parser *p) if ( (a = _loop1_97_rule(p)) // lambda_param_with_default+ && - (b = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (b = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_with_default+ lambda_star_etc?")); @@ -13978,7 +13978,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop0_104_rule(p)) // lambda_param_maybe_default* && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' lambda_param_no_default lambda_param_maybe_default* lambda_kwds?")); @@ -14011,7 +14011,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop1_105_rule(p)) // lambda_param_maybe_default+ && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' lambda_param_maybe_default+ lambda_kwds?")); @@ -14289,7 +14289,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' ) @@ -14318,7 +14318,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 11) // token=':' ) @@ -14479,7 +14479,7 @@ list_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_named_expressions_rule(p), 1) // star_named_expressions? + (a = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -14544,7 +14544,7 @@ tuple_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = _tmp_107_rule(p), 1) // [star_named_expression ',' star_named_expressions?] + (a = _tmp_107_rule(p), !p->error_indicator) // [star_named_expression ',' star_named_expressions?] && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -14674,7 +14674,7 @@ dict_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 25)) // token='{' && - (a = double_starred_kvpairs_rule(p), 1) // double_starred_kvpairs? + (a = double_starred_kvpairs_rule(p), !p->error_indicator) // double_starred_kvpairs? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -14755,7 +14755,7 @@ double_starred_kvpairs_rule(Parser *p) if ( (a = _gather_108_rule(p)) // ','.double_starred_kvpair+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ double_starred_kvpairs[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.double_starred_kvpair+ ','?")); @@ -15433,7 +15433,7 @@ arguments_rule(Parser *p) if ( (a = args_rule(p)) // args && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -15510,7 +15510,7 @@ args_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_114_rule(p)) // ','.(starred_expression | (assigment_expression | expression !':=') !'=')+ && - (b = _tmp_116_rule(p), 1) // [',' kwargs] + (b = _tmp_116_rule(p), !p->error_indicator) // [',' kwargs] ) { D(fprintf(stderr, "%*c+ args[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.(starred_expression | (assigment_expression | expression !':=') !'=')+ [',' kwargs]")); @@ -16013,7 +16013,7 @@ star_targets_rule(Parser *p) && (b = _loop0_125_rule(p)) // ((',' star_target))* && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))* ','?")); @@ -16067,7 +16067,7 @@ star_targets_list_seq_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_126_rule(p)) // ','.star_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_list_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_target+ ','?")); @@ -16115,7 +16115,7 @@ star_targets_tuple_seq_rule(Parser *p) && (b = _loop1_128_rule(p)) // ((',' star_target))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_tuple_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))+ ','?")); @@ -16478,7 +16478,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = star_targets_tuple_seq_rule(p), 1) // star_targets_tuple_seq? + (a = star_targets_tuple_seq_rule(p), !p->error_indicator) // star_targets_tuple_seq? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -16517,7 +16517,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_targets_list_seq_rule(p), 1) // star_targets_list_seq? + (a = star_targets_list_seq_rule(p), !p->error_indicator) // star_targets_list_seq? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -16950,7 +16950,7 @@ t_primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -17108,7 +17108,7 @@ del_targets_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_130_rule(p)) // ','.del_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ del_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.del_target+ ','?")); @@ -17354,7 +17354,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -17393,7 +17393,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -17824,7 +17824,7 @@ invalid_arguments_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (_opt_var = _tmp_141_rule(p), 1) // [args | expression for_if_clauses] + (_opt_var = _tmp_141_rule(p), !p->error_indicator) // [args | expression for_if_clauses] ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); @@ -19434,7 +19434,7 @@ invalid_for_target_rule(Parser *p) UNUSED(_opt_var); // Silence compiler warnings expr_ty a; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 630)) // token='for' && @@ -19612,7 +19612,7 @@ invalid_with_stmt_rule(Parser *p) void *_opt_var; UNUSED(_opt_var); // Silence compiler warnings if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 606)) // token='with' && @@ -19645,7 +19645,7 @@ invalid_with_stmt_rule(Parser *p) void *_opt_var_1; UNUSED(_opt_var_1); // Silence compiler warnings if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 606)) // token='with' && @@ -19653,7 +19653,7 @@ invalid_with_stmt_rule(Parser *p) && (_gather_164_var = _gather_164_rule(p)) // ','.(expressions ['as' star_target])+ && - (_opt_var_1 = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var_1 = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -19700,7 +19700,7 @@ invalid_with_stmt_indent_rule(Parser *p) Token * a; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 606)) // token='with' && @@ -19743,7 +19743,7 @@ invalid_with_stmt_indent_rule(Parser *p) Token * a; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 606)) // token='with' && @@ -19751,7 +19751,7 @@ invalid_with_stmt_indent_rule(Parser *p) && (_gather_168_var = _gather_168_rule(p)) // ','.(expressions ['as' star_target])+ && - (_opt_var_1 = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var_1 = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -19898,7 +19898,7 @@ invalid_except_stmt_rule(Parser *p) && (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_171_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_171_rule(p), !p->error_indicator) // ['as' NAME] && (_literal_1 = _PyPegen_expect_token(p, 11)) // token=':' ) @@ -19932,7 +19932,7 @@ invalid_except_stmt_rule(Parser *p) && (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_172_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_172_rule(p), !p->error_indicator) // ['as' NAME] && (newline_var = _PyPegen_expect_token(p, NEWLINE)) // token='NEWLINE' ) @@ -20062,7 +20062,7 @@ invalid_except_stmt_indent_rule(Parser *p) && (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_173_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_173_rule(p), !p->error_indicator) // ['as' NAME] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -20233,7 +20233,7 @@ invalid_case_block_rule(Parser *p) && (patterns_var = patterns_rule(p)) // patterns && - (_opt_var = guard_rule(p), 1) // guard? + (_opt_var = guard_rule(p), !p->error_indicator) // guard? && _PyPegen_lookahead_with_int(0, _PyPegen_expect_token, p, 11) // token=':' ) @@ -20268,7 +20268,7 @@ invalid_case_block_rule(Parser *p) && (patterns_var = patterns_rule(p)) // patterns && - (_opt_var = guard_rule(p), 1) // guard? + (_opt_var = guard_rule(p), !p->error_indicator) // guard? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -20446,7 +20446,7 @@ invalid_class_argument_pattern_rule(Parser *p) asdl_pattern_seq* a; asdl_seq* keyword_patterns_var; if ( - (_opt_var = _tmp_174_rule(p), 1) // [positional_patterns ','] + (_opt_var = _tmp_174_rule(p), !p->error_indicator) // [positional_patterns ','] && (keyword_patterns_var = keyword_patterns_rule(p)) // keyword_patterns && @@ -20801,7 +20801,7 @@ invalid_for_stmt_rule(Parser *p) expr_ty star_expressions_var; expr_ty star_targets_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 630)) // token='for' && @@ -20868,7 +20868,7 @@ invalid_def_raw_rule(Parser *p) expr_ty name_var; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 632)) // token='def' && @@ -20876,11 +20876,11 @@ invalid_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (_opt_var_1 = params_rule(p), 1) // params? + (_opt_var_1 = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (_opt_var_2 = _tmp_175_rule(p), 1) // ['->' expression] + (_opt_var_2 = _tmp_175_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_token(p, 11)) // token=':' && @@ -20936,7 +20936,7 @@ invalid_class_def_raw_rule(Parser *p) && (name_var = _PyPegen_name_token(p)) // NAME && - (_opt_var = _tmp_176_rule(p), 1) // ['(' arguments? ')'] + (_opt_var = _tmp_176_rule(p), !p->error_indicator) // ['(' arguments? ')'] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -23038,7 +23038,7 @@ _tmp_33_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (z = arguments_rule(p), 1) // arguments? + (z = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -26354,7 +26354,7 @@ _tmp_87_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (d = expression_rule(p), 1) // expression? + (d = expression_rule(p), !p->error_indicator) // expression? ) { D(fprintf(stderr, "%*c+ _tmp_87[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "':' expression?")); @@ -27708,7 +27708,7 @@ _tmp_107_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (z = star_named_expressions_rule(p), 1) // star_named_expressions? + (z = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? ) { D(fprintf(stderr, "%*c+ _tmp_107[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_named_expression ',' star_named_expressions?")); @@ -31699,7 +31699,7 @@ _tmp_176_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (_opt_var = arguments_rule(p), 1) // arguments? + (_opt_var = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -32708,7 +32708,7 @@ _tmp_197_rule(Parser *p) if ( (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_202_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_202_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_197[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ['as' star_target]")); @@ -32748,7 +32748,7 @@ _tmp_198_rule(Parser *p) if ( (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_203_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_203_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_198[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expressions ['as' star_target]")); @@ -32788,7 +32788,7 @@ _tmp_199_rule(Parser *p) if ( (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_204_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_204_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_199[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ['as' star_target]")); @@ -32828,7 +32828,7 @@ _tmp_200_rule(Parser *p) if ( (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_205_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_205_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_200[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expressions ['as' star_target]")); diff --git a/Parser/pegen.c b/Parser/pegen.c index e4d2692539264..b00eff3432dec 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -350,10 +350,18 @@ tokenizer_error(Parser *p) errtype = PyExc_IndentationError; msg = "too many levels of indentation"; break; - case E_LINECONT: - col_offset = strlen(strtok(p->tok->buf, "\n")) - 1; + case E_LINECONT: { + char* loc = strrchr(p->tok->buf, '\n'); + const char* last_char = p->tok->cur - 1; + if (loc != NULL && loc != last_char) { + col_offset = p->tok->cur - loc - 1; + p->tok->buf = loc; + } else { + col_offset = last_char - p->tok->buf - 1; + } msg = "unexpected character after line continuation character"; break; + } default: msg = "unknown parsing error"; } diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index 29c310fa76b17..10dc94b04fefd 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -88,7 +88,7 @@ def __str__(self) -> str: if self.arguments: parts.append(f"({', '.join(map(str, self.arguments))})") if self.force_true: - parts.append(", 1") + parts.append(", !p->error_indicator") if self.assigned_variable: if self.assigned_variable_type: parts = [ From webhook-mailer at python.org Tue Oct 19 16:11:21 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Tue, 19 Oct 2021 20:11:21 -0000 Subject: [Python-checkins] bpo-45506: Go back to not running most of test_embed in out-of-tree builds. (gh-29063) Message-ID: https://github.com/python/cpython/commit/f4b12440cf24d7636755aac5b2645e47713557c7 commit: f4b12440cf24d7636755aac5b2645e47713557c7 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-19T14:11:16-06:00 summary: bpo-45506: Go back to not running most of test_embed in out-of-tree builds. (gh-29063) In gh-28954 I adjusted how test_embed determines if it should be skipped. That broke out-of-tree builds. This change fixes them. https://bugs.python.org/issue45506 files: M Lib/test/test_embed.py diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 4b4396efb5cad..4186f011e2388 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -61,7 +61,7 @@ def setUp(self): else: exepath = os.path.join(builddir, 'Programs') self.test_exe = exe = os.path.join(exepath, exename) - if not os.path.exists(exe): + if exepath != support.REPO_ROOT or not os.path.exists(exe): self.skipTest("%r doesn't exist" % exe) # This is needed otherwise we get a fatal error: # "Py_Initialize: Unable to get the locale encoding From webhook-mailer at python.org Tue Oct 19 16:21:08 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 20:21:08 -0000 Subject: [Python-checkins] [3.10] bpo-45500: Rewrite test_dbm (GH-29002) (GH-29069) Message-ID: https://github.com/python/cpython/commit/325b2c223453203b2fa9ce7b9bcebdbef03adf70 commit: 325b2c223453203b2fa9ce7b9bcebdbef03adf70 branch: 3.10 author: ?ukasz Langa committer: ambv date: 2021-10-19T22:21:03+02:00 summary: [3.10] bpo-45500: Rewrite test_dbm (GH-29002) (GH-29069) * Generate test classes at import time. It allows to filter them when run with unittest. E.g: "./python -m unittest test.test_dbm.TestCase_gnu -v". * Create a database class in a new directory which will be removed after test. It guarantees that all created files and directories be removed and will not conflict with other dbm tests. * Restore dbm._defaultmod after tests. Previously it was set to the last dbm module (dbm.dumb) which affected other tests. * Enable the whichdb test for dbm.dumb. * Move test_keys to the correct test class. It does not test whichdb(). * Remove some outdated code and comments.. (cherry picked from commit 975b94b9de969777218e96a9950c1dab2dab65a0) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_dbm.py diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py index e02d1e16ae3da..16a023084e984 100644 --- a/Lib/test/test_dbm.py +++ b/Lib/test/test_dbm.py @@ -1,23 +1,21 @@ """Test script for the dbm.open function based on testdumbdbm.py""" import unittest -import glob +import dbm +import os from test.support import import_helper from test.support import os_helper -# Skip tests if dbm module doesn't exist. -dbm = import_helper.import_module('dbm') - try: from dbm import ndbm except ImportError: ndbm = None -_fname = os_helper.TESTFN +dirname = os_helper.TESTFN +_fname = os.path.join(dirname, os_helper.TESTFN) # -# Iterates over every database module supported by dbm currently available, -# setting dbm to use each in turn, and yielding that module +# Iterates over every database module supported by dbm currently available. # def dbm_iterator(): for name in dbm._names: @@ -31,11 +29,12 @@ def dbm_iterator(): # # Clean up all scratch databases we might have created during testing # -def delete_files(): - # we don't know the precise name the underlying database uses - # so we use glob to locate all names - for f in glob.glob(glob.escape(_fname) + "*"): - os_helper.unlink(f) +def cleaunup_test_dir(): + os_helper.rmtree(dirname) + +def setup_test_dir(): + cleaunup_test_dir() + os.mkdir(dirname) class AnyDBMTestCase: @@ -134,80 +133,67 @@ def read_helper(self, f): for key in self._dict: self.assertEqual(self._dict[key], f[key.encode("ascii")]) - def tearDown(self): - delete_files() + def test_keys(self): + with dbm.open(_fname, 'c') as d: + self.assertEqual(d.keys(), []) + a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] + for k, v in a: + d[k] = v + self.assertEqual(sorted(d.keys()), sorted(k for (k, v) in a)) + for k, v in a: + self.assertIn(k, d) + self.assertEqual(d[k], v) + self.assertNotIn(b'xxx', d) + self.assertRaises(KeyError, lambda: d[b'xxx']) def setUp(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) dbm._defaultmod = self.module - delete_files() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() class WhichDBTestCase(unittest.TestCase): def test_whichdb(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) for module in dbm_iterator(): # Check whether whichdb correctly guesses module name # for databases opened with "module" module. - # Try with empty files first name = module.__name__ - if name == 'dbm.dumb': - continue # whichdb can't support dbm.dumb - delete_files() - f = module.open(_fname, 'c') - f.close() + setup_test_dir() + dbm._defaultmod = module + # Try with empty files first + with module.open(_fname, 'c'): pass self.assertEqual(name, self.dbm.whichdb(_fname)) # Now add a key - f = module.open(_fname, 'w') - f[b"1"] = b"1" - # and test that we can find it - self.assertIn(b"1", f) - # and read it - self.assertEqual(f[b"1"], b"1") - f.close() + with module.open(_fname, 'w') as f: + f[b"1"] = b"1" + # and test that we can find it + self.assertIn(b"1", f) + # and read it + self.assertEqual(f[b"1"], b"1") self.assertEqual(name, self.dbm.whichdb(_fname)) @unittest.skipUnless(ndbm, reason='Test requires ndbm') def test_whichdb_ndbm(self): # Issue 17198: check that ndbm which is referenced in whichdb is defined - db_file = '{}_ndbm.db'.format(_fname) - with open(db_file, 'w'): - self.addCleanup(os_helper.unlink, db_file) - self.assertIsNone(self.dbm.whichdb(db_file[:-3])) - - def tearDown(self): - delete_files() + with open(_fname + '.db', 'wb'): pass + self.assertIsNone(self.dbm.whichdb(_fname)) def setUp(self): - delete_files() - self.filename = os_helper.TESTFN - self.d = dbm.open(self.filename, 'c') - self.d.close() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() self.dbm = import_helper.import_fresh_module('dbm') - def test_keys(self): - self.d = dbm.open(self.filename, 'c') - self.assertEqual(self.d.keys(), []) - a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] - for k, v in a: - self.d[k] = v - self.assertEqual(sorted(self.d.keys()), sorted(k for (k, v) in a)) - for k, v in a: - self.assertIn(k, self.d) - self.assertEqual(self.d[k], v) - self.assertNotIn(b'xxx', self.d) - self.assertRaises(KeyError, lambda: self.d[b'xxx']) - self.d.close() - - -def load_tests(loader, tests, pattern): - classes = [] - for mod in dbm_iterator(): - classes.append(type("TestCase-" + mod.__name__, - (AnyDBMTestCase, unittest.TestCase), - {'module': mod})) - suites = [unittest.makeSuite(c) for c in classes] - - tests.addTests(suites) - return tests + +for mod in dbm_iterator(): + assert mod.__name__.startswith('dbm.') + suffix = mod.__name__[4:] + testname = f'TestCase_{suffix}' + globals()[testname] = type(testname, + (AnyDBMTestCase, unittest.TestCase), + {'module': mod}) + if __name__ == "__main__": unittest.main() From webhook-mailer at python.org Tue Oct 19 16:30:34 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 20:30:34 -0000 Subject: [Python-checkins] bpo-39679: Add tests for classmethod/staticmethod singledispatchmethods (GH-29034) Message-ID: https://github.com/python/cpython/commit/ad6d162e518963711d24c80f1b7d6079bd437584 commit: ad6d162e518963711d24c80f1b7d6079bd437584 branch: main author: Alex Waygood committer: ambv date: 2021-10-19T22:30:27+02:00 summary: bpo-39679: Add tests for classmethod/staticmethod singledispatchmethods (GH-29034) In Python 3.8 and 3.9, stacking `@functools.singledispatchmethod` on top of `@classmethod` or `@staticmethod` caused an exception to be raised if the method was registered using type-annotations rather than `@method.register(int)`. This was not caught by unit tests, however, as the tests only tested the `@method.register(int)` way of registering additional implementations. The bug is no longer present in Python 3.10+, but `test_functools.py` is still lacking regression tests for these cases. This commit adds these test cases. files: A Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst M Lib/test/test_functools.py diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index fece8256a3e26..bdb4ddcc60cac 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -2437,6 +2437,48 @@ def _(self, arg: str): self.assertEqual(a.t(''), "str") self.assertEqual(a.t(0.0), "base") + def test_staticmethod_type_ann_register(self): + class A: + @functools.singledispatchmethod + @staticmethod + def t(arg): + return arg + @t.register + @staticmethod + def _(arg: int): + return isinstance(arg, int) + @t.register + @staticmethod + def _(arg: str): + return isinstance(arg, str) + a = A() + + self.assertTrue(A.t(0)) + self.assertTrue(A.t('')) + self.assertEqual(A.t(0.0), 0.0) + + def test_classmethod_type_ann_register(self): + class A: + def __init__(self, arg): + self.arg = arg + + @functools.singledispatchmethod + @classmethod + def t(cls, arg): + return cls("base") + @t.register + @classmethod + def _(cls, arg: int): + return cls("int") + @t.register + @classmethod + def _(cls, arg: str): + return cls("str") + + self.assertEqual(A.t(0).arg, "int") + self.assertEqual(A.t('').arg, "str") + self.assertEqual(A.t(0.0).arg, "base") + def test_invalid_registrations(self): msg_prefix = "Invalid first argument to `register()`: " msg_suffix = ( diff --git a/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst b/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst new file mode 100644 index 0000000000000..b0d1b686392ef --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst @@ -0,0 +1,2 @@ +Add more test cases for `@functools.singledispatchmethod` when combined with +`@classmethod` or `@staticmethod`. From webhook-mailer at python.org Tue Oct 19 16:31:23 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 20:31:23 -0000 Subject: [Python-checkins] [3.10] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (GH-29070) Message-ID: https://github.com/python/cpython/commit/5c9cab595e56aeb118bff77ece784dbac30b4338 commit: 5c9cab595e56aeb118bff77ece784dbac30b4338 branch: 3.10 author: ?ukasz Langa committer: ambv date: 2021-10-19T22:31:18+02:00 summary: [3.10] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (GH-29070) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing.. (cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57) Co-authored-by: Pablo Galindo Salgado files: A Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst M Lib/test/test_exceptions.py M Parser/parser.c M Parser/pegen.c M Tools/peg_generator/pegen/c_generator.py diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 4213dabfd8e71..4930c57d3fd7e 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -209,6 +209,10 @@ def check(self, src, lineno, offset, encoding='utf-8'): src = src.decode(encoding, 'replace') line = src.split('\n')[lineno-1] self.assertIn(line, cm.exception.text) + + def test_error_offset_continuation_characters(self): + check = self.check + check('"\\\n"(1 for c in I,\\\n\\', 2, 2) def testSyntaxErrorOffset(self): check = self.check diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst new file mode 100644 index 0000000000000..97e29813ab266 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst @@ -0,0 +1,2 @@ +Fix parser crash when reporting errors involving invalid continuation +characters. Patch by Pablo Galindo. diff --git a/Parser/parser.c b/Parser/parser.c index 2a437d5281b43..eccf104d679a8 100644 --- a/Parser/parser.c +++ b/Parser/parser.c @@ -948,7 +948,7 @@ file_rule(Parser *p) void *a; Token * endmarker_var; if ( - (a = statements_rule(p), 1) // statements? + (a = statements_rule(p), !p->error_indicator) // statements? && (endmarker_var = _PyPegen_expect_token(p, ENDMARKER)) // token='ENDMARKER' ) @@ -1087,7 +1087,7 @@ func_type_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = type_expressions_rule(p), 1) // type_expressions? + (a = type_expressions_rule(p), !p->error_indicator) // type_expressions? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -1689,7 +1689,7 @@ simple_stmts_rule(Parser *p) if ( (a = (asdl_stmt_seq*)_gather_12_rule(p)) // ';'.simple_stmt+ && - (_opt_var = _PyPegen_expect_token(p, 13), 1) // ';'? + (_opt_var = _PyPegen_expect_token(p, 13), !p->error_indicator) // ';'? && (newline_var = _PyPegen_expect_token(p, NEWLINE)) // token='NEWLINE' ) @@ -2309,7 +2309,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_19_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_19_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ':' expression ['=' annotated_rhs]")); @@ -2351,7 +2351,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_21_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_21_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "('(' single_target ')' | single_subscript_attribute_target) ':' expression ['=' annotated_rhs]")); @@ -2392,7 +2392,7 @@ assignment_rule(Parser *p) && _PyPegen_lookahead_with_int(0, _PyPegen_expect_token, p, 22) // token='=' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "((star_targets '='))+ (yield_expr | star_expressions) !'=' TYPE_COMMENT?")); @@ -3047,7 +3047,7 @@ assert_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_29_rule(p), 1) // [',' expression] + (b = _tmp_29_rule(p), !p->error_indicator) // [',' expression] ) { D(fprintf(stderr, "%*c+ assert_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'assert' expression [',' expression]")); @@ -3433,7 +3433,7 @@ import_from_targets_rule(Parser *p) && (a = import_from_as_names_rule(p)) // import_from_as_names && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -3602,7 +3602,7 @@ import_from_as_name_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_35_rule(p), 1) // ['as' NAME] + (b = _tmp_35_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ import_from_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ['as' NAME]")); @@ -3705,7 +3705,7 @@ dotted_as_name_rule(Parser *p) if ( (a = dotted_name_rule(p)) // dotted_name && - (b = _tmp_38_rule(p), 1) // ['as' NAME] + (b = _tmp_38_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ dotted_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "dotted_name ['as' NAME]")); @@ -3943,7 +3943,7 @@ if_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ if_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'if' named_expression ':' block else_block?")); @@ -4081,7 +4081,7 @@ elif_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ elif_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'elif' named_expression ':' block else_block?")); @@ -4237,7 +4237,7 @@ while_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ while_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'while' named_expression ':' block else_block?")); @@ -4339,11 +4339,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'for' star_targets 'in' ~ star_expressions &&':' TYPE_COMMENT? block else_block?")); @@ -4403,11 +4403,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "ASYNC 'for' star_targets 'in' ~ star_expressions &&':' TYPE_COMMENT? block else_block?")); @@ -4527,7 +4527,7 @@ with_stmt_rule(Parser *p) && (a = (asdl_withitem_seq*)_gather_39_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -4576,7 +4576,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -4627,7 +4627,7 @@ with_stmt_rule(Parser *p) && (a = (asdl_withitem_seq*)_gather_43_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -4679,7 +4679,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -4931,9 +4931,9 @@ try_stmt_rule(Parser *p) && (ex = (asdl_excepthandler_seq*)_loop1_48_rule(p)) // except_block+ && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? && - (f = finally_block_rule(p), 1) // finally_block? + (f = finally_block_rule(p), !p->error_indicator) // finally_block? ) { D(fprintf(stderr, "%*c+ try_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'try' &&':' block except_block+ else_block? finally_block?")); @@ -5023,7 +5023,7 @@ except_block_rule(Parser *p) && (e = expression_rule(p)) // expression && - (t = _tmp_49_rule(p), 1) // ['as' NAME] + (t = _tmp_49_rule(p), !p->error_indicator) // ['as' NAME] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -5314,7 +5314,7 @@ subject_expr_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (values = star_named_expressions_rule(p), 1) // star_named_expressions? + (values = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? ) { D(fprintf(stderr, "%*c+ subject_expr[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_named_expression ',' star_named_expressions?")); @@ -5410,7 +5410,7 @@ case_block_rule(Parser *p) && (pattern = patterns_rule(p)) // patterns && - (guard = guard_rule(p), 1) // guard? + (guard = guard_rule(p), !p->error_indicator) // guard? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -7163,7 +7163,7 @@ sequence_pattern_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (patterns = maybe_sequence_pattern_rule(p), 1) // maybe_sequence_pattern? + (patterns = maybe_sequence_pattern_rule(p), !p->error_indicator) // maybe_sequence_pattern? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -7202,7 +7202,7 @@ sequence_pattern_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (patterns = open_sequence_pattern_rule(p), 1) // open_sequence_pattern? + (patterns = open_sequence_pattern_rule(p), !p->error_indicator) // open_sequence_pattern? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -7260,7 +7260,7 @@ open_sequence_pattern_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (patterns = maybe_sequence_pattern_rule(p), 1) // maybe_sequence_pattern? + (patterns = maybe_sequence_pattern_rule(p), !p->error_indicator) // maybe_sequence_pattern? ) { D(fprintf(stderr, "%*c+ open_sequence_pattern[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "maybe_star_pattern ',' maybe_sequence_pattern?")); @@ -7305,7 +7305,7 @@ maybe_sequence_pattern_rule(Parser *p) if ( (patterns = _gather_57_rule(p)) // ','.maybe_star_pattern+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ maybe_sequence_pattern[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.maybe_star_pattern+ ','?")); @@ -7556,7 +7556,7 @@ mapping_pattern_rule(Parser *p) && (rest = double_star_pattern_rule(p)) // double_star_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -7605,7 +7605,7 @@ mapping_pattern_rule(Parser *p) && (rest = double_star_pattern_rule(p)) // double_star_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_2 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -7648,7 +7648,7 @@ mapping_pattern_rule(Parser *p) && (items = items_pattern_rule(p)) // items_pattern && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -7891,7 +7891,7 @@ class_pattern_rule(Parser *p) && (patterns = positional_patterns_rule(p)) // positional_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -7937,7 +7937,7 @@ class_pattern_rule(Parser *p) && (keywords = keyword_patterns_rule(p)) // keyword_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -7989,7 +7989,7 @@ class_pattern_rule(Parser *p) && (keywords = keyword_patterns_rule(p)) // keyword_patterns && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_2 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -8196,7 +8196,7 @@ return_stmt_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 500)) // token='return' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ return_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'return' star_expressions?")); @@ -8261,7 +8261,7 @@ raise_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_66_rule(p), 1) // ['from' expression] + (b = _tmp_66_rule(p), !p->error_indicator) // ['from' expression] ) { D(fprintf(stderr, "%*c+ raise_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'raise' expression ['from' expression]")); @@ -8452,15 +8452,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_67_rule(p), 1) // ['->' expression] + (a = _tmp_67_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -8512,15 +8512,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_68_rule(p), 1) // ['->' expression] + (a = _tmp_68_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -8728,7 +8728,7 @@ parameters_rule(Parser *p) && (c = _loop0_71_rule(p)) // param_with_default* && - (d = star_etc_rule(p), 1) // star_etc? + (d = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_no_default param_no_default* param_with_default* star_etc?")); @@ -8758,7 +8758,7 @@ parameters_rule(Parser *p) && (b = _loop0_72_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_with_default param_with_default* star_etc?")); @@ -8788,7 +8788,7 @@ parameters_rule(Parser *p) && (b = _loop0_74_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_no_default+ param_with_default* star_etc?")); @@ -8815,7 +8815,7 @@ parameters_rule(Parser *p) if ( (a = _loop1_75_rule(p)) // param_with_default+ && - (b = star_etc_rule(p), 1) // star_etc? + (b = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_with_default+ star_etc?")); @@ -9053,7 +9053,7 @@ star_etc_rule(Parser *p) && (b = _loop0_82_rule(p)) // param_maybe_default* && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' param_no_default param_maybe_default* kwds?")); @@ -9086,7 +9086,7 @@ star_etc_rule(Parser *p) && (b = _loop1_83_rule(p)) // param_maybe_default+ && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' param_maybe_default+ kwds?")); @@ -9220,7 +9220,7 @@ param_no_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_no_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param ',' TYPE_COMMENT?")); @@ -9247,7 +9247,7 @@ param_no_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -9299,7 +9299,7 @@ param_with_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_with_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default ',' TYPE_COMMENT?")); @@ -9329,7 +9329,7 @@ param_with_default_rule(Parser *p) && (c = default_rule(p)) // default && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -9379,11 +9379,11 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_maybe_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default? ',' TYPE_COMMENT?")); @@ -9411,9 +9411,9 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -9468,7 +9468,7 @@ param_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = annotation_rule(p), 1) // annotation? + (b = annotation_rule(p), !p->error_indicator) // annotation? ) { D(fprintf(stderr, "%*c+ param[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME annotation?")); @@ -9746,7 +9746,7 @@ class_def_raw_rule(Parser *p) && (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_85_rule(p), 1) // ['(' arguments? ')'] + (b = _tmp_85_rule(p), !p->error_indicator) // ['(' arguments? ')'] && (_literal = _PyPegen_expect_forced_token(p, 11, ":")) // forced_token=':' && @@ -9912,7 +9912,7 @@ star_expressions_rule(Parser *p) && (b = _loop1_86_rule(p)) // ((',' star_expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_expression ((',' star_expression))+ ','?")); @@ -10107,7 +10107,7 @@ star_named_expressions_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_87_rule(p)) // ','.star_named_expression+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_named_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_named_expression+ ','?")); @@ -10448,7 +10448,7 @@ expressions_rule(Parser *p) && (b = _loop1_89_rule(p)) // ((',' expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ((',' expression))+ ','?")); @@ -10704,7 +10704,7 @@ lambdef_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 528)) // token='lambda' && - (a = lambda_params_rule(p), 1) // lambda_params? + (a = lambda_params_rule(p), !p->error_indicator) // lambda_params? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -10827,7 +10827,7 @@ lambda_parameters_rule(Parser *p) && (c = _loop0_91_rule(p)) // lambda_param_with_default* && - (d = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (d = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_no_default lambda_param_no_default* lambda_param_with_default* lambda_star_etc?")); @@ -10857,7 +10857,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_92_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_with_default lambda_param_with_default* lambda_star_etc?")); @@ -10887,7 +10887,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_94_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_no_default+ lambda_param_with_default* lambda_star_etc?")); @@ -10914,7 +10914,7 @@ lambda_parameters_rule(Parser *p) if ( (a = _loop1_95_rule(p)) // lambda_param_with_default+ && - (b = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (b = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_with_default+ lambda_star_etc?")); @@ -11154,7 +11154,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop0_102_rule(p)) // lambda_param_maybe_default* && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' lambda_param_no_default lambda_param_maybe_default* lambda_kwds?")); @@ -11187,7 +11187,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop1_103_rule(p)) // lambda_param_maybe_default+ && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' lambda_param_maybe_default+ lambda_kwds?")); @@ -11465,7 +11465,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' ) @@ -11494,7 +11494,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 11) // token=':' ) @@ -14024,7 +14024,7 @@ primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -14176,7 +14176,7 @@ slices_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_108_rule(p)) // ','.slice+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ slices[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.slice+ ','?")); @@ -14238,13 +14238,13 @@ slice_rule(Parser *p) void *b; void *c; if ( - (a = expression_rule(p), 1) // expression? + (a = expression_rule(p), !p->error_indicator) // expression? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (b = expression_rule(p), 1) // expression? + (b = expression_rule(p), !p->error_indicator) // expression? && - (c = _tmp_110_rule(p), 1) // [':' expression?] + (c = _tmp_110_rule(p), !p->error_indicator) // [':' expression?] ) { D(fprintf(stderr, "%*c+ slice[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression? ':' expression? [':' expression?]")); @@ -14667,7 +14667,7 @@ list_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_named_expressions_rule(p), 1) // star_named_expressions? + (a = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -14819,7 +14819,7 @@ tuple_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = _tmp_115_rule(p), 1) // [star_named_expression ',' star_named_expressions?] + (a = _tmp_115_rule(p), !p->error_indicator) // [star_named_expression ',' star_named_expressions?] && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -15191,7 +15191,7 @@ dict_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 25)) // token='{' && - (a = double_starred_kvpairs_rule(p), 1) // double_starred_kvpairs? + (a = double_starred_kvpairs_rule(p), !p->error_indicator) // double_starred_kvpairs? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -15359,7 +15359,7 @@ double_starred_kvpairs_rule(Parser *p) if ( (a = _gather_118_rule(p)) // ','.double_starred_kvpair+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ double_starred_kvpairs[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.double_starred_kvpair+ ','?")); @@ -15730,7 +15730,7 @@ yield_expr_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 504)) // token='yield' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ yield_expr[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'yield' star_expressions?")); @@ -15788,7 +15788,7 @@ arguments_rule(Parser *p) if ( (a = args_rule(p)) // args && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -15865,7 +15865,7 @@ args_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_123_rule(p)) // ','.(starred_expression | (assigment_expression | expression !':=') !'=')+ && - (b = _tmp_125_rule(p), 1) // [',' kwargs] + (b = _tmp_125_rule(p), !p->error_indicator) // [',' kwargs] ) { D(fprintf(stderr, "%*c+ args[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.(starred_expression | (assigment_expression | expression !':=') !'=')+ [',' kwargs]")); @@ -16368,7 +16368,7 @@ star_targets_rule(Parser *p) && (b = _loop0_134_rule(p)) // ((',' star_target))* && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))* ','?")); @@ -16422,7 +16422,7 @@ star_targets_list_seq_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_135_rule(p)) // ','.star_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_list_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_target+ ','?")); @@ -16470,7 +16470,7 @@ star_targets_tuple_seq_rule(Parser *p) && (b = _loop1_137_rule(p)) // ((',' star_target))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_tuple_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))+ ','?")); @@ -16833,7 +16833,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = star_targets_tuple_seq_rule(p), 1) // star_targets_tuple_seq? + (a = star_targets_tuple_seq_rule(p), !p->error_indicator) // star_targets_tuple_seq? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -16872,7 +16872,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_targets_list_seq_rule(p), 1) // star_targets_list_seq? + (a = star_targets_list_seq_rule(p), !p->error_indicator) // star_targets_list_seq? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -17131,7 +17131,7 @@ del_targets_rule(Parser *p) if ( (a = (asdl_expr_seq*)_gather_139_rule(p)) // ','.del_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ del_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.del_target+ ','?")); @@ -17377,7 +17377,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -17416,7 +17416,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -17646,7 +17646,7 @@ t_primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -17846,7 +17846,7 @@ invalid_arguments_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (_opt_var = _tmp_141_rule(p), 1) // [args | expression for_if_clauses] + (_opt_var = _tmp_141_rule(p), !p->error_indicator) // [args | expression for_if_clauses] ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); @@ -19501,7 +19501,7 @@ invalid_for_target_rule(Parser *p) UNUSED(_opt_var); // Silence compiler warnings expr_ty a; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 517)) // token='for' && @@ -19679,7 +19679,7 @@ invalid_with_stmt_rule(Parser *p) void *_opt_var; UNUSED(_opt_var); // Silence compiler warnings if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 519)) // token='with' && @@ -19712,7 +19712,7 @@ invalid_with_stmt_rule(Parser *p) void *_opt_var_1; UNUSED(_opt_var_1); // Silence compiler warnings if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 519)) // token='with' && @@ -19720,7 +19720,7 @@ invalid_with_stmt_rule(Parser *p) && (_gather_164_var = _gather_164_rule(p)) // ','.(expressions ['as' star_target])+ && - (_opt_var_1 = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var_1 = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -19767,7 +19767,7 @@ invalid_with_stmt_indent_rule(Parser *p) Token * a; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 519)) // token='with' && @@ -19810,7 +19810,7 @@ invalid_with_stmt_indent_rule(Parser *p) Token * a; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 519)) // token='with' && @@ -19818,7 +19818,7 @@ invalid_with_stmt_indent_rule(Parser *p) && (_gather_168_var = _gather_168_rule(p)) // ','.(expressions ['as' star_target])+ && - (_opt_var_1 = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var_1 = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -19965,7 +19965,7 @@ invalid_except_stmt_rule(Parser *p) && (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_171_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_171_rule(p), !p->error_indicator) // ['as' NAME] && (_literal_1 = _PyPegen_expect_token(p, 11)) // token=':' ) @@ -19999,7 +19999,7 @@ invalid_except_stmt_rule(Parser *p) && (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_172_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_172_rule(p), !p->error_indicator) // ['as' NAME] && (newline_var = _PyPegen_expect_token(p, NEWLINE)) // token='NEWLINE' ) @@ -20129,7 +20129,7 @@ invalid_except_stmt_indent_rule(Parser *p) && (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_173_rule(p), 1) // ['as' NAME] + (_opt_var = _tmp_173_rule(p), !p->error_indicator) // ['as' NAME] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -20300,7 +20300,7 @@ invalid_case_block_rule(Parser *p) && (patterns_var = patterns_rule(p)) // patterns && - (_opt_var = guard_rule(p), 1) // guard? + (_opt_var = guard_rule(p), !p->error_indicator) // guard? && _PyPegen_lookahead_with_int(0, _PyPegen_expect_token, p, 11) // token=':' ) @@ -20335,7 +20335,7 @@ invalid_case_block_rule(Parser *p) && (patterns_var = patterns_rule(p)) // patterns && - (_opt_var = guard_rule(p), 1) // guard? + (_opt_var = guard_rule(p), !p->error_indicator) // guard? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -20513,7 +20513,7 @@ invalid_class_argument_pattern_rule(Parser *p) asdl_pattern_seq* a; asdl_seq* keyword_patterns_var; if ( - (_opt_var = _tmp_174_rule(p), 1) // [positional_patterns ','] + (_opt_var = _tmp_174_rule(p), !p->error_indicator) // [positional_patterns ','] && (keyword_patterns_var = keyword_patterns_rule(p)) // keyword_patterns && @@ -20868,7 +20868,7 @@ invalid_for_stmt_rule(Parser *p) expr_ty star_expressions_var; expr_ty star_targets_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 517)) // token='for' && @@ -20935,7 +20935,7 @@ invalid_def_raw_rule(Parser *p) expr_ty name_var; Token * newline_var; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (a = _PyPegen_expect_token(p, 526)) // token='def' && @@ -20943,11 +20943,11 @@ invalid_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (_opt_var_1 = params_rule(p), 1) // params? + (_opt_var_1 = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (_opt_var_2 = _tmp_175_rule(p), 1) // ['->' expression] + (_opt_var_2 = _tmp_175_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_token(p, 11)) // token=':' && @@ -21003,7 +21003,7 @@ invalid_class_def_raw_rule(Parser *p) && (name_var = _PyPegen_name_token(p)) // NAME && - (_opt_var = _tmp_176_rule(p), 1) // ['(' arguments? ')'] + (_opt_var = _tmp_176_rule(p), !p->error_indicator) // ['(' arguments? ')'] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -26246,7 +26246,7 @@ _tmp_85_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (z = arguments_rule(p), 1) // arguments? + (z = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -27875,7 +27875,7 @@ _tmp_110_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (d = expression_rule(p), 1) // expression? + (d = expression_rule(p), !p->error_indicator) // expression? ) { D(fprintf(stderr, "%*c+ _tmp_110[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "':' expression?")); @@ -28215,7 +28215,7 @@ _tmp_115_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (z = star_named_expressions_rule(p), 1) // star_named_expressions? + (z = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? ) { D(fprintf(stderr, "%*c+ _tmp_115[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_named_expression ',' star_named_expressions?")); @@ -31766,7 +31766,7 @@ _tmp_176_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (_opt_var = arguments_rule(p), 1) // arguments? + (_opt_var = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -32775,7 +32775,7 @@ _tmp_197_rule(Parser *p) if ( (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_202_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_202_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_197[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ['as' star_target]")); @@ -32815,7 +32815,7 @@ _tmp_198_rule(Parser *p) if ( (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_203_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_203_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_198[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expressions ['as' star_target]")); @@ -32855,7 +32855,7 @@ _tmp_199_rule(Parser *p) if ( (expression_var = expression_rule(p)) // expression && - (_opt_var = _tmp_204_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_204_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_199[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ['as' star_target]")); @@ -32895,7 +32895,7 @@ _tmp_200_rule(Parser *p) if ( (expressions_var = expressions_rule(p)) // expressions && - (_opt_var = _tmp_205_rule(p), 1) // ['as' star_target] + (_opt_var = _tmp_205_rule(p), !p->error_indicator) // ['as' star_target] ) { D(fprintf(stderr, "%*c+ _tmp_200[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expressions ['as' star_target]")); diff --git a/Parser/pegen.c b/Parser/pegen.c index 1bb975d684a0d..66e4b1929711f 100644 --- a/Parser/pegen.c +++ b/Parser/pegen.c @@ -371,10 +371,18 @@ tokenizer_error(Parser *p) errtype = PyExc_IndentationError; msg = "too many levels of indentation"; break; - case E_LINECONT: - col_offset = strlen(strtok(p->tok->buf, "\n")) - 1; + case E_LINECONT: { + char* loc = strrchr(p->tok->buf, '\n'); + const char* last_char = p->tok->cur - 1; + if (loc != NULL && loc != last_char) { + col_offset = p->tok->cur - loc - 1; + p->tok->buf = loc; + } else { + col_offset = last_char - p->tok->buf - 1; + } msg = "unexpected character after line continuation character"; break; + } default: msg = "unknown parsing error"; } diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index 7941978252d17..3079edf54b46a 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -88,7 +88,7 @@ def __str__(self) -> str: if self.arguments: parts.append(f"({', '.join(map(str, self.arguments))})") if self.force_true: - parts.append(", 1") + parts.append(", !p->error_indicator") if self.assigned_variable: if self.assigned_variable_type: parts = ["(", self.assigned_variable, " = ", '(', self.assigned_variable_type, ')', *parts, ")"] From webhook-mailer at python.org Tue Oct 19 18:07:18 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 22:07:18 -0000 Subject: [Python-checkins] bpo-39679: Add tests for classmethod/staticmethod singledispatchmethods (GH-29034) (GH-29072) Message-ID: https://github.com/python/cpython/commit/c15ba304f35362470e29ea5626fed28366bc9571 commit: c15ba304f35362470e29ea5626fed28366bc9571 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T00:07:13+02:00 summary: bpo-39679: Add tests for classmethod/staticmethod singledispatchmethods (GH-29034) (GH-29072) In Python 3.8 and 3.9, stacking `@functools.singledispatchmethod` on top of `@classmethod` or `@staticmethod` caused an exception to be raised if the method was registered using type-annotations rather than `@method.register(int)`. This was not caught by unit tests, however, as the tests only tested the `@method.register(int)` way of registering additional implementations. The bug is no longer present in Python 3.10+, but `test_functools.py` is still lacking regression tests for these cases. This commit adds these test cases. (cherry picked from commit ad6d162e518963711d24c80f1b7d6079bd437584) Co-authored-by: Alex Waygood files: A Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst M Lib/test/test_functools.py diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index fece8256a3e26..bdb4ddcc60cac 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -2437,6 +2437,48 @@ def _(self, arg: str): self.assertEqual(a.t(''), "str") self.assertEqual(a.t(0.0), "base") + def test_staticmethod_type_ann_register(self): + class A: + @functools.singledispatchmethod + @staticmethod + def t(arg): + return arg + @t.register + @staticmethod + def _(arg: int): + return isinstance(arg, int) + @t.register + @staticmethod + def _(arg: str): + return isinstance(arg, str) + a = A() + + self.assertTrue(A.t(0)) + self.assertTrue(A.t('')) + self.assertEqual(A.t(0.0), 0.0) + + def test_classmethod_type_ann_register(self): + class A: + def __init__(self, arg): + self.arg = arg + + @functools.singledispatchmethod + @classmethod + def t(cls, arg): + return cls("base") + @t.register + @classmethod + def _(cls, arg: int): + return cls("int") + @t.register + @classmethod + def _(cls, arg: str): + return cls("str") + + self.assertEqual(A.t(0).arg, "int") + self.assertEqual(A.t('').arg, "str") + self.assertEqual(A.t(0.0).arg, "base") + def test_invalid_registrations(self): msg_prefix = "Invalid first argument to `register()`: " msg_suffix = ( diff --git a/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst b/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst new file mode 100644 index 0000000000000..b0d1b686392ef --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-18-16-18-41.bpo-39679.F18qcE.rst @@ -0,0 +1,2 @@ +Add more test cases for `@functools.singledispatchmethod` when combined with +`@classmethod` or `@staticmethod`. From webhook-mailer at python.org Tue Oct 19 18:19:39 2021 From: webhook-mailer at python.org (ambv) Date: Tue, 19 Oct 2021 22:19:39 -0000 Subject: [Python-checkins] bpo-25625: add contextlib.chdir (GH-28271) Message-ID: https://github.com/python/cpython/commit/3592980f9122ab0d9ed93711347742d110b749c2 commit: 3592980f9122ab0d9ed93711347742d110b749c2 branch: main author: Filipe La?ns committer: ambv date: 2021-10-20T00:19:27+02:00 summary: bpo-25625: add contextlib.chdir (GH-28271) Added non parallel-safe :func:`~contextlib.chdir` context manager to change the current working directory and then restore it on exit. Simple wrapper around :func:`~os.chdir`. Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-09-10-12-53-28.bpo-25625.SzcBCw.rst M Doc/library/contextlib.rst M Lib/contextlib.py M Lib/test/test_contextlib.py diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index bc38a63a52d97..ae0ee7232a10c 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -353,6 +353,23 @@ Functions and classes provided: .. versionadded:: 3.5 +.. function:: chdir(path) + + Non parallel-safe context manager to change the current working directory. + As this changes a global state, the working directory, it is not suitable + for use in most threaded or aync contexts. It is also not suitable for most + non-linear code execution, like generators, where the program execution is + temporarily relinquished -- unless explicitely desired, you should not yield + when this context manager is active. + + This is a simple wrapper around :func:`~os.chdir`, it changes the current + working directory upon entering and restores the old one on exit. + + This context manager is :ref:`reentrant `. + + .. versionadded:: 3.11 + + .. class:: ContextDecorator() A base class that enables a context manager to also be used as a decorator. @@ -900,8 +917,8 @@ but may also be used *inside* a :keyword:`!with` statement that is already using the same context manager. :class:`threading.RLock` is an example of a reentrant context manager, as are -:func:`suppress` and :func:`redirect_stdout`. Here's a very simple example of -reentrant use:: +:func:`suppress`, :func:`redirect_stdout`, and :func:`chdir`. Here's a very +simple example of reentrant use:: >>> from contextlib import redirect_stdout >>> from io import StringIO diff --git a/Lib/contextlib.py b/Lib/contextlib.py index d90ca5d8ef988..ee72258505714 100644 --- a/Lib/contextlib.py +++ b/Lib/contextlib.py @@ -1,5 +1,6 @@ """Utilities for with-statement contexts. See PEP 343.""" import abc +import os import sys import _collections_abc from collections import deque @@ -9,7 +10,8 @@ __all__ = ["asynccontextmanager", "contextmanager", "closing", "nullcontext", "AbstractContextManager", "AbstractAsyncContextManager", "AsyncExitStack", "ContextDecorator", "ExitStack", - "redirect_stdout", "redirect_stderr", "suppress", "aclosing"] + "redirect_stdout", "redirect_stderr", "suppress", "aclosing", + "chdir"] class AbstractContextManager(abc.ABC): @@ -762,3 +764,18 @@ async def __aenter__(self): async def __aexit__(self, *excinfo): pass + + +class chdir(AbstractContextManager): + """Non thread-safe context manager to change the current working directory.""" + + def __init__(self, path): + self.path = path + self._old_cwd = [] + + def __enter__(self): + self._old_cwd.append(os.getcwd()) + os.chdir(self.path) + + def __exit__(self, *excinfo): + os.chdir(self._old_cwd.pop()) diff --git a/Lib/test/test_contextlib.py b/Lib/test/test_contextlib.py index 7982d9d835a2b..bc8e4e4e2918f 100644 --- a/Lib/test/test_contextlib.py +++ b/Lib/test/test_contextlib.py @@ -1,6 +1,7 @@ """Unit tests for contextlib.py, and other context managers.""" import io +import os import sys import tempfile import threading @@ -1114,5 +1115,47 @@ def test_cm_is_reentrant(self): 1/0 self.assertTrue(outer_continued) + +class TestChdir(unittest.TestCase): + def test_simple(self): + old_cwd = os.getcwd() + target = os.path.join(os.path.dirname(__file__), 'data') + self.assertNotEqual(old_cwd, target) + + with chdir(target): + self.assertEqual(os.getcwd(), target) + self.assertEqual(os.getcwd(), old_cwd) + + def test_reentrant(self): + old_cwd = os.getcwd() + target1 = os.path.join(os.path.dirname(__file__), 'data') + target2 = os.path.join(os.path.dirname(__file__), 'ziptestdata') + self.assertNotIn(old_cwd, (target1, target2)) + chdir1, chdir2 = chdir(target1), chdir(target2) + + with chdir1: + self.assertEqual(os.getcwd(), target1) + with chdir2: + self.assertEqual(os.getcwd(), target2) + with chdir1: + self.assertEqual(os.getcwd(), target1) + self.assertEqual(os.getcwd(), target2) + self.assertEqual(os.getcwd(), target1) + self.assertEqual(os.getcwd(), old_cwd) + + def test_exception(self): + old_cwd = os.getcwd() + target = os.path.join(os.path.dirname(__file__), 'data') + self.assertNotEqual(old_cwd, target) + + try: + with chdir(target): + self.assertEqual(os.getcwd(), target) + raise RuntimeError("boom") + except RuntimeError as re: + self.assertEqual(str(re), "boom") + self.assertEqual(os.getcwd(), old_cwd) + + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-09-10-12-53-28.bpo-25625.SzcBCw.rst b/Misc/NEWS.d/next/Library/2021-09-10-12-53-28.bpo-25625.SzcBCw.rst new file mode 100644 index 0000000000000..c001683b657f5 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-10-12-53-28.bpo-25625.SzcBCw.rst @@ -0,0 +1,3 @@ +Added non parallel-safe :func:`~contextlib.chdir` context manager to change +the current working directory and then restore it on exit. Simple wrapper +around :func:`~os.chdir`. From webhook-mailer at python.org Tue Oct 19 19:16:45 2021 From: webhook-mailer at python.org (markshannon) Date: Tue, 19 Oct 2021 23:16:45 -0000 Subject: [Python-checkins] bpo-44525: Specialize ``CALL_FUNCTION`` for C function calls (GH-26934) Message-ID: https://github.com/python/cpython/commit/3163e68c342434db37c69669017f96a4bb2d5f13 commit: 3163e68c342434db37c69669017f96a4bb2d5f13 branch: main author: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> committer: markshannon date: 2021-10-20T00:16:36+01:00 summary: bpo-44525: Specialize ``CALL_FUNCTION`` for C function calls (GH-26934) files: A Misc/NEWS.d/next/Core and Builtins/2021-06-28-22-23-59.bpo-44525.sSvUKG.rst M Include/internal/pycore_code.h M Include/opcode.h M Lib/opcode.py M Python/ceval.c M Python/opcode_targets.h M Python/specialize.c diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index a91209b39f654..d464f3d2a8131 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -309,6 +309,7 @@ int _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *na int _Py_Specialize_BinarySubscr(PyObject *sub, PyObject *container, _Py_CODEUNIT *instr); int _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr); int _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *instr); +int _Py_Specialize_CallFunction(PyObject *callable, _Py_CODEUNIT *instr, int nargs, SpecializedCacheEntry *cache, PyObject *builtins); #define PRINT_SPECIALIZATION_STATS 0 #define PRINT_SPECIALIZATION_STATS_DETAILED 0 diff --git a/Include/opcode.h b/Include/opcode.h index fabb8d123bb9c..22d968ee0d4c7 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -148,29 +148,34 @@ extern "C" { #define BINARY_SUBSCR_LIST_INT 40 #define BINARY_SUBSCR_TUPLE_INT 41 #define BINARY_SUBSCR_DICT 42 -#define JUMP_ABSOLUTE_QUICK 43 -#define LOAD_ATTR_ADAPTIVE 44 -#define LOAD_ATTR_INSTANCE_VALUE 45 -#define LOAD_ATTR_WITH_HINT 46 -#define LOAD_ATTR_SLOT 47 -#define LOAD_ATTR_MODULE 48 -#define LOAD_GLOBAL_ADAPTIVE 58 -#define LOAD_GLOBAL_MODULE 80 -#define LOAD_GLOBAL_BUILTIN 81 -#define LOAD_METHOD_ADAPTIVE 87 -#define LOAD_METHOD_CACHED 88 -#define LOAD_METHOD_CLASS 120 -#define LOAD_METHOD_MODULE 122 -#define LOAD_METHOD_NO_DICT 123 -#define STORE_ATTR_ADAPTIVE 127 -#define STORE_ATTR_INSTANCE_VALUE 128 -#define STORE_ATTR_SLOT 134 -#define STORE_ATTR_WITH_HINT 140 -#define LOAD_FAST__LOAD_FAST 143 -#define STORE_FAST__LOAD_FAST 149 -#define LOAD_FAST__LOAD_CONST 150 -#define LOAD_CONST__LOAD_FAST 151 -#define STORE_FAST__STORE_FAST 153 +#define CALL_FUNCTION_ADAPTIVE 43 +#define CALL_FUNCTION_BUILTIN_O 44 +#define CALL_FUNCTION_BUILTIN_FAST 45 +#define CALL_FUNCTION_LEN 46 +#define CALL_FUNCTION_ISINSTANCE 47 +#define JUMP_ABSOLUTE_QUICK 48 +#define LOAD_ATTR_ADAPTIVE 58 +#define LOAD_ATTR_INSTANCE_VALUE 80 +#define LOAD_ATTR_WITH_HINT 81 +#define LOAD_ATTR_SLOT 87 +#define LOAD_ATTR_MODULE 88 +#define LOAD_GLOBAL_ADAPTIVE 120 +#define LOAD_GLOBAL_MODULE 122 +#define LOAD_GLOBAL_BUILTIN 123 +#define LOAD_METHOD_ADAPTIVE 127 +#define LOAD_METHOD_CACHED 128 +#define LOAD_METHOD_CLASS 134 +#define LOAD_METHOD_MODULE 140 +#define LOAD_METHOD_NO_DICT 143 +#define STORE_ATTR_ADAPTIVE 149 +#define STORE_ATTR_INSTANCE_VALUE 150 +#define STORE_ATTR_SLOT 151 +#define STORE_ATTR_WITH_HINT 153 +#define LOAD_FAST__LOAD_FAST 154 +#define STORE_FAST__LOAD_FAST 158 +#define LOAD_FAST__LOAD_CONST 159 +#define LOAD_CONST__LOAD_FAST 167 +#define STORE_FAST__STORE_FAST 168 #define DO_TRACING 255 #ifdef NEED_OPCODE_JUMP_TABLES static uint32_t _PyOpcode_RelativeJump[8] = { diff --git a/Lib/opcode.py b/Lib/opcode.py index 9d9f35855e25b..fe6066fc517a6 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -232,6 +232,11 @@ def jabs_op(name, op): "BINARY_SUBSCR_LIST_INT", "BINARY_SUBSCR_TUPLE_INT", "BINARY_SUBSCR_DICT", + "CALL_FUNCTION_ADAPTIVE", + "CALL_FUNCTION_BUILTIN_O", + "CALL_FUNCTION_BUILTIN_FAST", + "CALL_FUNCTION_LEN", + "CALL_FUNCTION_ISINSTANCE", "JUMP_ABSOLUTE_QUICK", "LOAD_ATTR_ADAPTIVE", "LOAD_ATTR_INSTANCE_VALUE", diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-28-22-23-59.bpo-44525.sSvUKG.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-28-22-23-59.bpo-44525.sSvUKG.rst new file mode 100644 index 0000000000000..8963d028cf8a9 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-06-28-22-23-59.bpo-44525.sSvUKG.rst @@ -0,0 +1,10 @@ +Setup initial specialization infrastructure for the ``CALL_FUNCTION`` opcode. +Implemented initial specializations for C function calls: + +* ``CALL_FUNCTION_BUILTIN_O`` for ``METH_O`` flag. + +* ``CALL_FUNCTION_BUILTIN_FAST`` for ``METH_FASTCALL`` flag without keywords. + +* ``CALL_FUNCTION_LEN`` for ``len(o)``. + +* ``CALL_FUNCTION_ISINSTANCE`` for ``isinstance(o, t)``. diff --git a/Python/ceval.c b/Python/ceval.c index 2fb60a84a112d..76325903149cd 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4660,6 +4660,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(CALL_FUNCTION) { PREDICTED(CALL_FUNCTION); + STAT_INC(CALL_FUNCTION, unquickened); PyObject *function; nargs = oparg; kwnames = NULL; @@ -4717,6 +4718,151 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } + TARGET(CALL_FUNCTION_ADAPTIVE) { + SpecializedCacheEntry *cache = GET_CACHE(); + if (cache->adaptive.counter == 0) { + next_instr--; + int nargs = cache->adaptive.original_oparg; + if (_Py_Specialize_CallFunction( + PEEK(nargs + 1), next_instr, nargs, cache, BUILTINS()) < 0) { + goto error; + } + DISPATCH(); + } + else { + STAT_INC(CALL_FUNCTION, deferred); + cache->adaptive.counter--; + oparg = cache->adaptive.original_oparg; + JUMP_TO_INSTRUCTION(CALL_FUNCTION); + } + } + + TARGET(CALL_FUNCTION_BUILTIN_O) { + assert(cframe.use_tracing == 0); + /* Builtin METH_O functions */ + + PyObject *callable = SECOND(); + DEOPT_IF(!PyCFunction_CheckExact(callable), CALL_FUNCTION); + DEOPT_IF(PyCFunction_GET_FLAGS(callable) != METH_O, CALL_FUNCTION); + _PyAdaptiveEntry *cache0 = &GET_CACHE()[0].adaptive; + record_cache_hit(cache0); + STAT_INC(CALL_FUNCTION, hit); + + PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable); + PyObject *arg = POP(); + PyObject *res = cfunc(PyCFunction_GET_SELF(callable), arg); + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + + /* Clear the stack of the function object. */ + Py_DECREF(arg); + Py_DECREF(callable); + SET_TOP(res); + if (res == NULL) { + goto error; + } + DISPATCH(); + } + + TARGET(CALL_FUNCTION_BUILTIN_FAST) { + assert(cframe.use_tracing == 0); + /* Builtin METH_FASTCALL functions, without keywords */ + SpecializedCacheEntry *caches = GET_CACHE(); + _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + int nargs = cache0->original_oparg; + PyObject **pfunc = &PEEK(nargs + 1); + PyObject *callable = *pfunc; + DEOPT_IF(!PyCFunction_CheckExact(callable), CALL_FUNCTION); + DEOPT_IF(PyCFunction_GET_FLAGS(callable) != METH_FASTCALL, + CALL_FUNCTION); + record_cache_hit(cache0); + STAT_INC(CALL_FUNCTION, hit); + + PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable); + /* res = func(self, args, nargs) */ + PyObject *res = ((_PyCFunctionFast)(void(*)(void))cfunc)( + PyCFunction_GET_SELF(callable), + &PEEK(nargs), + nargs); + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + + /* Clear the stack of the function object. */ + while (stack_pointer > pfunc) { + PyObject *x = POP(); + Py_DECREF(x); + } + PUSH(res); + if (res == NULL) { + /* Not deopting because this doesn't mean our optimization was + wrong. `res` can be NULL for valid reasons. Eg. getattr(x, + 'invalid'). In those cases an exception is set, so we must + handle it. + */ + goto error; + } + DISPATCH(); + } + + TARGET(CALL_FUNCTION_LEN) { + assert(cframe.use_tracing == 0); + /* len(o) */ + SpecializedCacheEntry *caches = GET_CACHE(); + _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + _PyObjectCache *cache1 = &caches[-1].obj; + assert(cache0->original_oparg == 1); + + PyObject *callable = SECOND(); + DEOPT_IF(callable != cache1->obj, CALL_FUNCTION); + record_cache_hit(cache0); + STAT_INC(CALL_FUNCTION, hit); + + Py_ssize_t len_i = PyObject_Length(TOP()); + if (len_i < 0) { + goto error; + } + PyObject *res = PyLong_FromSsize_t(len_i); + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + + /* Clear the stack of the function object. */ + Py_DECREF(POP()); + Py_DECREF(callable); + SET_TOP(res); + if (res == NULL) { + goto error; + } + DISPATCH(); + } + + TARGET(CALL_FUNCTION_ISINSTANCE) { + assert(cframe.use_tracing == 0); + /* isinstance(o, o2) */ + SpecializedCacheEntry *caches = GET_CACHE(); + _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + _PyObjectCache *cache1 = &caches[-1].obj; + assert(cache0->original_oparg == 2); + + PyObject *callable = THIRD(); + DEOPT_IF(callable != cache1->obj, CALL_FUNCTION); + record_cache_hit(cache0); + STAT_INC(CALL_FUNCTION, hit); + + int retval = PyObject_IsInstance(SECOND(), TOP()); + if (retval < 0) { + goto error; + } + PyObject *res = PyBool_FromLong(retval); + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + + /* Clear the stack of the function object. */ + Py_DECREF(POP()); + Py_DECREF(POP()); + Py_DECREF(callable); + SET_TOP(res); + if (res == NULL) { + goto error; + } + DISPATCH(); + } + TARGET(CALL_FUNCTION_EX) { PREDICTED(CALL_FUNCTION_EX); PyObject *func, *callargs, *kwargs = NULL, *result; @@ -4985,6 +5131,7 @@ MISS_WITH_CACHE(LOAD_ATTR) MISS_WITH_CACHE(STORE_ATTR) MISS_WITH_CACHE(LOAD_GLOBAL) MISS_WITH_CACHE(LOAD_METHOD) +MISS_WITH_CACHE(CALL_FUNCTION) MISS_WITH_OPARG_COUNTER(BINARY_SUBSCR) MISS_WITH_OPARG_COUNTER(BINARY_ADD) MISS_WITH_OPARG_COUNTER(BINARY_MULTIPLY) diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 7d701e81a9a48..4179689e8a866 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -42,12 +42,12 @@ static void *opcode_targets[256] = { &&TARGET_BINARY_SUBSCR_LIST_INT, &&TARGET_BINARY_SUBSCR_TUPLE_INT, &&TARGET_BINARY_SUBSCR_DICT, + &&TARGET_CALL_FUNCTION_ADAPTIVE, + &&TARGET_CALL_FUNCTION_BUILTIN_O, + &&TARGET_CALL_FUNCTION_BUILTIN_FAST, + &&TARGET_CALL_FUNCTION_LEN, + &&TARGET_CALL_FUNCTION_ISINSTANCE, &&TARGET_JUMP_ABSOLUTE_QUICK, - &&TARGET_LOAD_ATTR_ADAPTIVE, - &&TARGET_LOAD_ATTR_INSTANCE_VALUE, - &&TARGET_LOAD_ATTR_WITH_HINT, - &&TARGET_LOAD_ATTR_SLOT, - &&TARGET_LOAD_ATTR_MODULE, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, @@ -57,7 +57,7 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_ADD, &&TARGET_INPLACE_SUBTRACT, &&TARGET_INPLACE_MULTIPLY, - &&TARGET_LOAD_GLOBAL_ADAPTIVE, + &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_INPLACE_MODULO, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, @@ -79,15 +79,15 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_AND, &&TARGET_INPLACE_XOR, &&TARGET_INPLACE_OR, - &&TARGET_LOAD_GLOBAL_MODULE, - &&TARGET_LOAD_GLOBAL_BUILTIN, + &&TARGET_LOAD_ATTR_INSTANCE_VALUE, + &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, &&TARGET_SETUP_ANNOTATIONS, &&TARGET_YIELD_VALUE, - &&TARGET_LOAD_METHOD_ADAPTIVE, - &&TARGET_LOAD_METHOD_CACHED, + &&TARGET_LOAD_ATTR_SLOT, + &&TARGET_LOAD_ATTR_MODULE, &&TARGET_POP_EXCEPT, &&TARGET_STORE_NAME, &&TARGET_DELETE_NAME, @@ -119,46 +119,46 @@ static void *opcode_targets[256] = { &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, - &&TARGET_LOAD_METHOD_CLASS, + &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_JUMP_IF_NOT_EXC_MATCH, - &&TARGET_LOAD_METHOD_MODULE, - &&TARGET_LOAD_METHOD_NO_DICT, + &&TARGET_LOAD_GLOBAL_MODULE, + &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, - &&TARGET_STORE_ATTR_ADAPTIVE, - &&TARGET_STORE_ATTR_INSTANCE_VALUE, + &&TARGET_LOAD_METHOD_ADAPTIVE, + &&TARGET_LOAD_METHOD_CACHED, &&TARGET_GEN_START, &&TARGET_RAISE_VARARGS, &&TARGET_CALL_FUNCTION, &&TARGET_MAKE_FUNCTION, &&TARGET_BUILD_SLICE, - &&TARGET_STORE_ATTR_SLOT, + &&TARGET_LOAD_METHOD_CLASS, &&TARGET_MAKE_CELL, &&TARGET_LOAD_CLOSURE, &&TARGET_LOAD_DEREF, &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, - &&TARGET_STORE_ATTR_WITH_HINT, + &&TARGET_LOAD_METHOD_MODULE, &&TARGET_CALL_FUNCTION_KW, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_LOAD_FAST__LOAD_FAST, + &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, - &&TARGET_STORE_FAST__LOAD_FAST, - &&TARGET_LOAD_FAST__LOAD_CONST, - &&TARGET_LOAD_CONST__LOAD_FAST, + &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_STORE_ATTR_INSTANCE_VALUE, + &&TARGET_STORE_ATTR_SLOT, &&TARGET_MATCH_CLASS, - &&TARGET_STORE_FAST__STORE_FAST, - &&_unknown_opcode, + &&TARGET_STORE_ATTR_WITH_HINT, + &&TARGET_LOAD_FAST__LOAD_FAST, &&TARGET_FORMAT_VALUE, &&TARGET_BUILD_CONST_KEY_MAP, &&TARGET_BUILD_STRING, - &&_unknown_opcode, - &&_unknown_opcode, + &&TARGET_STORE_FAST__LOAD_FAST, + &&TARGET_LOAD_FAST__LOAD_CONST, &&TARGET_LOAD_METHOD, &&TARGET_CALL_METHOD, &&TARGET_LIST_EXTEND, @@ -166,8 +166,8 @@ static void *opcode_targets[256] = { &&TARGET_DICT_MERGE, &&TARGET_DICT_UPDATE, &&TARGET_CALL_METHOD_KW, - &&_unknown_opcode, - &&_unknown_opcode, + &&TARGET_LOAD_CONST__LOAD_FAST, + &&TARGET_STORE_FAST__STORE_FAST, &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, diff --git a/Python/specialize.c b/Python/specialize.c index 264637dc96856..ee573d29a474e 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -128,6 +128,7 @@ _Py_GetSpecializationStats(void) { err += add_stat_dict(stats, BINARY_MULTIPLY, "binary_multiply"); err += add_stat_dict(stats, BINARY_SUBSCR, "binary_subscr"); err += add_stat_dict(stats, STORE_ATTR, "store_attr"); + err += add_stat_dict(stats, CALL_FUNCTION, "call_function"); if (err < 0) { Py_DECREF(stats); return NULL; @@ -185,6 +186,7 @@ _Py_PrintSpecializationStats(void) print_stats(out, &_specialization_stats[BINARY_MULTIPLY], "binary_multiply"); print_stats(out, &_specialization_stats[BINARY_SUBSCR], "binary_subscr"); print_stats(out, &_specialization_stats[STORE_ATTR], "store_attr"); + print_stats(out, &_specialization_stats[CALL_FUNCTION], "call_function"); if (out != stderr) { fclose(out); } @@ -235,6 +237,7 @@ static uint8_t adaptive_opcodes[256] = { [BINARY_ADD] = BINARY_ADD_ADAPTIVE, [BINARY_MULTIPLY] = BINARY_MULTIPLY_ADAPTIVE, [BINARY_SUBSCR] = BINARY_SUBSCR_ADAPTIVE, + [CALL_FUNCTION] = CALL_FUNCTION_ADAPTIVE, [STORE_ATTR] = STORE_ATTR_ADAPTIVE, }; @@ -246,6 +249,7 @@ static uint8_t cache_requirements[256] = { [BINARY_ADD] = 0, [BINARY_MULTIPLY] = 0, [BINARY_SUBSCR] = 0, + [CALL_FUNCTION] = 2, /* _PyAdaptiveEntry and _PyObjectCache */ [STORE_ATTR] = 2, /* _PyAdaptiveEntry and _PyAttrCache */ }; @@ -457,6 +461,15 @@ _Py_Quicken(PyCodeObject *code) { #define SPEC_FAIL_NON_FUNCTION_SCOPE 11 #define SPEC_FAIL_DIFFERENT_TYPES 12 +/* Call function */ + +#define SPEC_FAIL_PYCFUNCTION 10 +#define SPEC_FAIL_PYCFUNCTION_WITH_KEYWORDS 13 +#define SPEC_FAIL_PYCFUNCTION_FAST_WITH_KEYWORDS 14 +#define SPEC_FAIL_PYCFUNCTION_NOARGS 15 +#define SPEC_FAIL_BAD_CALL_FLAGS 16 +#define SPEC_FAIL_PYTHON_FUNCTION 17 +#define SPEC_FAIL_IMMUTABLE_CLASS 18 static int specialize_module_load_attr( @@ -1222,3 +1235,137 @@ _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *ins assert(!PyErr_Occurred()); return 0; } + +#if COLLECT_SPECIALIZATION_STATS_DETAILED +static int +builtin_call_fail_kind(int ml_flags) +{ + switch (ml_flags & (METH_VARARGS | METH_FASTCALL | METH_NOARGS | METH_O | + METH_KEYWORDS | METH_METHOD)) { + case METH_VARARGS: + return SPEC_FAIL_PYCFUNCTION; + case METH_VARARGS | METH_KEYWORDS: + return SPEC_FAIL_PYCFUNCTION_WITH_KEYWORDS; + case METH_FASTCALL | METH_KEYWORDS: + return SPEC_FAIL_PYCFUNCTION_FAST_WITH_KEYWORDS; + case METH_NOARGS: + return SPEC_FAIL_PYCFUNCTION_NOARGS; + /* This case should never happen with PyCFunctionObject -- only + PyMethodObject. See zlib.compressobj()'s methods for an example. + */ + case METH_METHOD | METH_FASTCALL | METH_KEYWORDS: + default: + return SPEC_FAIL_BAD_CALL_FLAGS; + } +} +#endif + +static int +specialize_c_call(PyObject *callable, _Py_CODEUNIT *instr, int nargs, + SpecializedCacheEntry *cache, PyObject *builtins) +{ + _PyObjectCache *cache1 = &cache[-1].obj; + if (PyCFunction_GET_FUNCTION(callable) == NULL) { + return 1; + } + switch (PyCFunction_GET_FLAGS(callable) & + (METH_VARARGS | METH_FASTCALL | METH_NOARGS | METH_O | + METH_KEYWORDS | METH_METHOD)) { + case METH_O: { + if (nargs != 1) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_OUT_OF_RANGE); + return 1; + } + /* len(o) */ + PyObject *builtin_len = PyDict_GetItemString(builtins, "len"); + if (callable == builtin_len) { + cache1->obj = builtin_len; // borrowed + *instr = _Py_MAKECODEUNIT(CALL_FUNCTION_LEN, + _Py_OPARG(*instr)); + return 0; + } + *instr = _Py_MAKECODEUNIT(CALL_FUNCTION_BUILTIN_O, + _Py_OPARG(*instr)); + return 0; + } + case METH_FASTCALL: { + if (nargs == 2) { + /* isinstance(o1, o2) */ + PyObject *builtin_isinstance = PyDict_GetItemString( + builtins, "isinstance"); + if (callable == builtin_isinstance) { + cache1->obj = builtin_isinstance; // borrowed + *instr = _Py_MAKECODEUNIT(CALL_FUNCTION_ISINSTANCE, + _Py_OPARG(*instr)); + return 0; + } + } + *instr = _Py_MAKECODEUNIT(CALL_FUNCTION_BUILTIN_FAST, + _Py_OPARG(*instr)); + return 0; + } + default: + SPECIALIZATION_FAIL(CALL_FUNCTION, + builtin_call_fail_kind(PyCFunction_GET_FLAGS(callable))); + return 1; + } +} + +#if COLLECT_SPECIALIZATION_STATS_DETAILED +static int +call_fail_kind(PyObject *callable) +{ + if (PyFunction_Check(callable)) { + return SPEC_FAIL_PYTHON_FUNCTION; + } + // new-style bound methods + else if (PyInstanceMethod_Check(callable)) { + return SPEC_FAIL_METHOD; + } + else if (PyMethod_Check(callable)) { + return SPEC_FAIL_METHOD; + } + // builtin method + else if (PyCMethod_Check(callable)) { + return SPEC_FAIL_METHOD; + } + else if (PyType_Check(callable)) { + PyTypeObject *type = Py_TYPE(callable); + return PyType_HasFeature(type, Py_TPFLAGS_IMMUTABLETYPE) ? + SPEC_FAIL_IMMUTABLE_CLASS : SPEC_FAIL_MUTABLE_CLASS; + } + return SPEC_FAIL_OTHER; +} +#endif + +/* TODO: + - Specialize calling types. + - Specialize python function calls. +*/ +int +_Py_Specialize_CallFunction( + PyObject *callable, _Py_CODEUNIT *instr, + int nargs, SpecializedCacheEntry *cache, + PyObject *builtins) +{ + int fail; + if (PyCFunction_CheckExact(callable)) { + fail = specialize_c_call(callable, instr, nargs, cache, builtins); + } + else { + SPECIALIZATION_FAIL(CALL_FUNCTION, call_fail_kind(callable)); + fail = 1; + } + _PyAdaptiveEntry *cache0 = &cache->adaptive; + if (fail) { + STAT_INC(CALL_FUNCTION, specialization_failure); + assert(!PyErr_Occurred()); + cache_backoff(cache0); + } + else { + STAT_INC(CALL_FUNCTION, specialization_success); + assert(!PyErr_Occurred()); + cache0->counter = saturating_start(); + } + return 0; +} From webhook-mailer at python.org Tue Oct 19 22:54:51 2021 From: webhook-mailer at python.org (zware) Date: Wed, 20 Oct 2021 02:54:51 -0000 Subject: [Python-checkins] bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) Message-ID: https://github.com/python/cpython/commit/4fe454c6f54b0948af67b53af6c2f35af6377e69 commit: 4fe454c6f54b0948af67b53af6c2f35af6377e69 branch: main author: Zachary Ware committer: zware date: 2021-10-19T21:54:19-05:00 summary: bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) Since v8.6.11, a few configuration options seem to accept an empty value where they did not previously; particularly the `type` of a `Menu` widget, and the `compound` of any ttk widget with a label. Providing an explicit expected error message to `checkEnumParam` bypasses the check of an empty value, which no longer raises `TclError`. files: M Lib/tkinter/test/test_tkinter/test_widgets.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index 39334de8cf41c..cc227e579679b 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -1241,8 +1241,11 @@ def test_configure_title(self): def test_configure_type(self): widget = self.create() - self.checkEnumParam(widget, 'type', - 'normal', 'tearoff', 'menubar') + self.checkEnumParam( + widget, 'type', + 'normal', 'tearoff', 'menubar', + errmsg='bad type "{}": must be normal, tearoff, or menubar', + ) def test_entryconfigure(self): m1 = self.create() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index a2a4de2e5cc41..904aed0ac2df6 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -169,10 +169,13 @@ def checkImageParam(self, widget, name): errmsg='image "spam" doesn\'t exist') def test_configure_compound(self): + options = 'none text image center top bottom left right'.split() + errmsg = ( + 'bad compound "{}": must be' + f' {", ".join(options[:-1])}, or {options[-1]}' + ) widget = self.create() - self.checkEnumParam(widget, 'compound', - 'none', 'text', 'image', 'center', - 'top', 'bottom', 'left', 'right') + self.checkEnumParam(widget, 'compound', *options, errmsg=errmsg) def test_configure_state(self): widget = self.create() From webhook-mailer at python.org Tue Oct 19 23:34:32 2021 From: webhook-mailer at python.org (zware) Date: Wed, 20 Oct 2021 03:34:32 -0000 Subject: [Python-checkins] bpo-38371: Remove remaining use of tk.split from bigmem tcl test (GH-29082) Message-ID: https://github.com/python/cpython/commit/085ccb0f177988065dbe9ef4c5cda434560066bc commit: 085ccb0f177988065dbe9ef4c5cda434560066bc branch: main author: Zachary Ware committer: zware date: 2021-10-19T22:34:23-05:00 summary: bpo-38371: Remove remaining use of tk.split from bigmem tcl test (GH-29082) files: M Lib/test/test_tcl.py diff --git a/Lib/test/test_tcl.py b/Lib/test/test_tcl.py index f18baa54e4a01..581c31ccb72fd 100644 --- a/Lib/test/test_tcl.py +++ b/Lib/test/test_tcl.py @@ -729,7 +729,6 @@ def test_huge_string_builtins(self, size): self.assertRaises(OverflowError, tk.exprlong, value) self.assertRaises(OverflowError, tk.exprboolean, value) self.assertRaises(OverflowError, tk.splitlist, value) - self.assertRaises(OverflowError, tk.split, value) self.assertRaises(OverflowError, tk.createcommand, value, max) self.assertRaises(OverflowError, tk.deletecommand, value) From webhook-mailer at python.org Wed Oct 20 04:06:55 2021 From: webhook-mailer at python.org (corona10) Date: Wed, 20 Oct 2021 08:06:55 -0000 Subject: [Python-checkins] [3.9] bpo-45500: Rewrite test_dbm (GH-29002) (GH-29074) Message-ID: https://github.com/python/cpython/commit/d46b2217d13bbcf8145da8c12a487ba775d6f162 commit: d46b2217d13bbcf8145da8c12a487ba775d6f162 branch: 3.9 author: ?ukasz Langa committer: corona10 date: 2021-10-20T17:06:38+09:00 summary: [3.9] bpo-45500: Rewrite test_dbm (GH-29002) (GH-29074) * Generate test classes at import time. It allows to filter them when run with unittest. E.g: "./python -m unittest test.test_dbm.TestCase_gnu -v". * Create a database class in a new directory which will be removed after test. It guarantees that all created files and directories be removed and will not conflict with other dbm tests. * Restore dbm._defaultmod after tests. Previously it was set to the last dbm module (dbm.dumb) which affected other tests. * Enable the whichdb test for dbm.dumb. * Move test_keys to the correct test class. It does not test whichdb(). * Remove some outdated code and comments.. (cherry picked from commit 975b94b9de969777218e96a9950c1dab2dab65a0) Co-authored-by: Serhiy Storchaka Co-authored-by: Serhiy Storchaka files: M Lib/test/test_dbm.py diff --git a/Lib/test/test_dbm.py b/Lib/test/test_dbm.py index 571da973aab0e..eb40d95142e4e 100644 --- a/Lib/test/test_dbm.py +++ b/Lib/test/test_dbm.py @@ -1,22 +1,20 @@ """Test script for the dbm.open function based on testdumbdbm.py""" import unittest -import glob +import dbm +import os import test.support -# Skip tests if dbm module doesn't exist. -dbm = test.support.import_module('dbm') - try: from dbm import ndbm except ImportError: ndbm = None -_fname = test.support.TESTFN +dirname = test.support.TESTFN +_fname = os.path.join(dirname, test.support.TESTFN) # -# Iterates over every database module supported by dbm currently available, -# setting dbm to use each in turn, and yielding that module +# Iterates over every database module supported by dbm currently available. # def dbm_iterator(): for name in dbm._names: @@ -30,11 +28,12 @@ def dbm_iterator(): # # Clean up all scratch databases we might have created during testing # -def delete_files(): - # we don't know the precise name the underlying database uses - # so we use glob to locate all names - for f in glob.glob(glob.escape(_fname) + "*"): - test.support.unlink(f) +def cleaunup_test_dir(): + test.support.rmtree(dirname) + +def setup_test_dir(): + cleaunup_test_dir() + os.mkdir(dirname) class AnyDBMTestCase: @@ -133,80 +132,67 @@ def read_helper(self, f): for key in self._dict: self.assertEqual(self._dict[key], f[key.encode("ascii")]) - def tearDown(self): - delete_files() + def test_keys(self): + with dbm.open(_fname, 'c') as d: + self.assertEqual(d.keys(), []) + a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] + for k, v in a: + d[k] = v + self.assertEqual(sorted(d.keys()), sorted(k for (k, v) in a)) + for k, v in a: + self.assertIn(k, d) + self.assertEqual(d[k], v) + self.assertNotIn(b'xxx', d) + self.assertRaises(KeyError, lambda: d[b'xxx']) def setUp(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) dbm._defaultmod = self.module - delete_files() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() class WhichDBTestCase(unittest.TestCase): def test_whichdb(self): + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) for module in dbm_iterator(): # Check whether whichdb correctly guesses module name # for databases opened with "module" module. - # Try with empty files first name = module.__name__ - if name == 'dbm.dumb': - continue # whichdb can't support dbm.dumb - delete_files() - f = module.open(_fname, 'c') - f.close() + setup_test_dir() + dbm._defaultmod = module + # Try with empty files first + with module.open(_fname, 'c'): pass self.assertEqual(name, self.dbm.whichdb(_fname)) # Now add a key - f = module.open(_fname, 'w') - f[b"1"] = b"1" - # and test that we can find it - self.assertIn(b"1", f) - # and read it - self.assertEqual(f[b"1"], b"1") - f.close() + with module.open(_fname, 'w') as f: + f[b"1"] = b"1" + # and test that we can find it + self.assertIn(b"1", f) + # and read it + self.assertEqual(f[b"1"], b"1") self.assertEqual(name, self.dbm.whichdb(_fname)) @unittest.skipUnless(ndbm, reason='Test requires ndbm') def test_whichdb_ndbm(self): # Issue 17198: check that ndbm which is referenced in whichdb is defined - db_file = '{}_ndbm.db'.format(_fname) - with open(db_file, 'w'): - self.addCleanup(test.support.unlink, db_file) - self.assertIsNone(self.dbm.whichdb(db_file[:-3])) - - def tearDown(self): - delete_files() + with open(_fname + '.db', 'wb'): pass + self.assertIsNone(self.dbm.whichdb(_fname)) def setUp(self): - delete_files() - self.filename = test.support.TESTFN - self.d = dbm.open(self.filename, 'c') - self.d.close() + self.addCleanup(cleaunup_test_dir) + setup_test_dir() self.dbm = test.support.import_fresh_module('dbm') - def test_keys(self): - self.d = dbm.open(self.filename, 'c') - self.assertEqual(self.d.keys(), []) - a = [(b'a', b'b'), (b'12345678910', b'019237410982340912840198242')] - for k, v in a: - self.d[k] = v - self.assertEqual(sorted(self.d.keys()), sorted(k for (k, v) in a)) - for k, v in a: - self.assertIn(k, self.d) - self.assertEqual(self.d[k], v) - self.assertNotIn(b'xxx', self.d) - self.assertRaises(KeyError, lambda: self.d[b'xxx']) - self.d.close() - - -def load_tests(loader, tests, pattern): - classes = [] - for mod in dbm_iterator(): - classes.append(type("TestCase-" + mod.__name__, - (AnyDBMTestCase, unittest.TestCase), - {'module': mod})) - suites = [unittest.makeSuite(c) for c in classes] - - tests.addTests(suites) - return tests + +for mod in dbm_iterator(): + assert mod.__name__.startswith('dbm.') + suffix = mod.__name__[4:] + testname = f'TestCase_{suffix}' + globals()[testname] = type(testname, + (AnyDBMTestCase, unittest.TestCase), + {'module': mod}) + if __name__ == "__main__": unittest.main() From webhook-mailer at python.org Wed Oct 20 05:32:18 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 09:32:18 -0000 Subject: [Python-checkins] bpo-45474: Exclude all of marshal.h if Py_LIMITED_API is defined (GH-29061) Message-ID: https://github.com/python/cpython/commit/98fa3b53e2aecc5ecec64a921bc9cf4f9d07ac75 commit: 98fa3b53e2aecc5ecec64a921bc9cf4f9d07ac75 branch: main author: Petr Viktorin committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T02:32:14-07:00 summary: bpo-45474: Exclude all of marshal.h if Py_LIMITED_API is defined (GH-29061) Also, reword the What's New messages: this doesn't change the limited API, it only brings the Py_LIMITED_API macro closer to the ideal of only allowing the limited API. Automerge-Triggered-By: GH:encukou files: M Doc/whatsnew/3.11.rst M Include/marshal.h M Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b583c2f31c4cc..13c1e72306653 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -627,14 +627,17 @@ Removed ``Py_IS_INFINITY()`` macro. (Contributed by Victor Stinner in :issue:`45440`.) -* Remove two functions from the limited C API: +* The following items are no longer available when :c:macro:`Py_LIMITED_API` + is defined: * :c:func:`PyMarshal_WriteLongToFile` * :c:func:`PyMarshal_WriteObjectToFile` + * :c:func:`PyMarshal_ReadObjectFromString` + * :c:func:`PyMarshal_WriteObjectToString` + * the ``Py_MARSHAL_VERSION`` macro - The :pep:`384` excludes functions expecting ``FILE*`` from the stable ABI. + These are not part of the :ref:`limited API `. - Remove also the ``Py_MARSHAL_VERSION`` macro from the limited C API. (Contributed by Victor Stinner in :issue:`45474`.) * Exclude :c:func:`PyWeakref_GET_OBJECT` from the limited C API. It never diff --git a/Include/marshal.h b/Include/marshal.h index 36ef6a779ec2c..f8b0de80cfc38 100644 --- a/Include/marshal.h +++ b/Include/marshal.h @@ -3,6 +3,8 @@ #ifndef Py_MARSHAL_H #define Py_MARSHAL_H +#ifndef Py_LIMITED_API + #ifdef __cplusplus extern "C" { #endif @@ -11,7 +13,6 @@ PyAPI_FUNC(PyObject *) PyMarshal_ReadObjectFromString(const char *, Py_ssize_t); PyAPI_FUNC(PyObject *) PyMarshal_WriteObjectToString(PyObject *, int); -#ifndef Py_LIMITED_API #define Py_MARSHAL_VERSION 4 PyAPI_FUNC(long) PyMarshal_ReadLongFromFile(FILE *); @@ -21,9 +22,10 @@ PyAPI_FUNC(PyObject *) PyMarshal_ReadLastObjectFromFile(FILE *); PyAPI_FUNC(void) PyMarshal_WriteLongToFile(long, FILE *, int); PyAPI_FUNC(void) PyMarshal_WriteObjectToFile(PyObject *, FILE *, int); -#endif #ifdef __cplusplus } #endif + +#endif /* Py_LIMITED_API */ #endif /* !Py_MARSHAL_H */ diff --git a/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst b/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst index 90bf498579c11..d41f1b72b8966 100644 --- a/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst +++ b/Misc/NEWS.d/next/C API/2021-10-14-22-16-56.bpo-45474.1OkJQh.rst @@ -1,10 +1,11 @@ -Remove two functions from the limited C API: +The following items are no longer available when ``Py_LIMITED_API`` is defined: * :c:func:`PyMarshal_WriteLongToFile` * :c:func:`PyMarshal_WriteObjectToFile` +* :c:func:`PyMarshal_ReadObjectFromString` +* :c:func:`PyMarshal_WriteObjectToString` +* the ``Py_MARSHAL_VERSION`` macro -The :pep:`384` excludes functions expecting ``FILE*`` from the stable ABI. - -Remove also the ``Py_MARSHAL_VERSION`` macro from the limited C API. +These are not part of the :ref:`limited API `. Patch by Victor Stinner. From webhook-mailer at python.org Wed Oct 20 09:50:36 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 13:50:36 -0000 Subject: [Python-checkins] bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29081) Message-ID: https://github.com/python/cpython/commit/15cd7a7f9edcc121f273ae1d8007128282f8f85f commit: 15cd7a7f9edcc121f273ae1d8007128282f8f85f branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T15:50:31+02:00 summary: bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29081) Since v8.6.11, a few configuration options seem to accept an empty value where they did not previously; particularly the `type` of a `Menu` widget, and the `compound` of any ttk widget with a label. Providing an explicit expected error message to `checkEnumParam` bypasses the check of an empty value, which no longer raises `TclError`. (cherry picked from commit 4fe454c6f54b0948af67b53af6c2f35af6377e69) Co-authored-by: Zachary Ware files: M Lib/tkinter/test/test_tkinter/test_widgets.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index 39334de8cf41c..cc227e579679b 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -1241,8 +1241,11 @@ def test_configure_title(self): def test_configure_type(self): widget = self.create() - self.checkEnumParam(widget, 'type', - 'normal', 'tearoff', 'menubar') + self.checkEnumParam( + widget, 'type', + 'normal', 'tearoff', 'menubar', + errmsg='bad type "{}": must be normal, tearoff, or menubar', + ) def test_entryconfigure(self): m1 = self.create() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index a2a4de2e5cc41..904aed0ac2df6 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -169,10 +169,13 @@ def checkImageParam(self, widget, name): errmsg='image "spam" doesn\'t exist') def test_configure_compound(self): + options = 'none text image center top bottom left right'.split() + errmsg = ( + 'bad compound "{}": must be' + f' {", ".join(options[:-1])}, or {options[-1]}' + ) widget = self.create() - self.checkEnumParam(widget, 'compound', - 'none', 'text', 'image', 'center', - 'top', 'bottom', 'left', 'right') + self.checkEnumParam(widget, 'compound', *options, errmsg=errmsg) def test_configure_state(self): widget = self.create() From webhook-mailer at python.org Wed Oct 20 09:51:12 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 13:51:12 -0000 Subject: [Python-checkins] bpo-25625: [doc] fix async/aync typo (GH-29091) Message-ID: https://github.com/python/cpython/commit/a774285e7d3e63c24dbaaee0b0d9e59ded5c49bf commit: a774285e7d3e63c24dbaaee0b0d9e59ded5c49bf branch: main author: Thomas Grainger committer: ambv date: 2021-10-20T15:51:07+02:00 summary: bpo-25625: [doc] fix async/aync typo (GH-29091) files: M Doc/library/contextlib.rst diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index ae0ee7232a10c..a800a0b8dee4b 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -357,7 +357,7 @@ Functions and classes provided: Non parallel-safe context manager to change the current working directory. As this changes a global state, the working directory, it is not suitable - for use in most threaded or aync contexts. It is also not suitable for most + for use in most threaded or async contexts. It is also not suitable for most non-linear code execution, like generators, where the program execution is temporarily relinquished -- unless explicitely desired, you should not yield when this context manager is active. From webhook-mailer at python.org Wed Oct 20 10:08:52 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 14:08:52 -0000 Subject: [Python-checkins] bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29093) Message-ID: https://github.com/python/cpython/commit/2ce38167000fef3a71f1783acdda2a2cf7a2df39 commit: 2ce38167000fef3a71f1783acdda2a2cf7a2df39 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T16:08:43+02:00 summary: bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29093) Since v8.6.11, a few configuration options seem to accept an empty value where they did not previously; particularly the `type` of a `Menu` widget, and the `compound` of any ttk widget with a label. Providing an explicit expected error message to `checkEnumParam` bypasses the check of an empty value, which no longer raises `TclError`. (cherry picked from commit 4fe454c6f54b0948af67b53af6c2f35af6377e69) Co-authored-by: Zachary Ware files: M Lib/tkinter/test/test_tkinter/test_widgets.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index cbb5d634e3d34..e4c155efe77d7 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -1244,8 +1244,11 @@ def test_configure_title(self): def test_configure_type(self): widget = self.create() - self.checkEnumParam(widget, 'type', - 'normal', 'tearoff', 'menubar') + self.checkEnumParam( + widget, 'type', + 'normal', 'tearoff', 'menubar', + errmsg='bad type "{}": must be normal, tearoff, or menubar', + ) def test_entryconfigure(self): m1 = self.create() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 1fac83a004a6d..88a800d05e4ca 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -169,10 +169,13 @@ def checkImageParam(self, widget, name): errmsg='image "spam" doesn\'t exist') def test_configure_compound(self): + options = 'none text image center top bottom left right'.split() + errmsg = ( + 'bad compound "{}": must be' + f' {", ".join(options[:-1])}, or {options[-1]}' + ) widget = self.create() - self.checkEnumParam(widget, 'compound', - 'none', 'text', 'image', 'center', - 'top', 'bottom', 'left', 'right') + self.checkEnumParam(widget, 'compound', *options, errmsg=errmsg) def test_configure_state(self): widget = self.create() From webhook-mailer at python.org Wed Oct 20 10:31:03 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 14:31:03 -0000 Subject: [Python-checkins] bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29080) Message-ID: https://github.com/python/cpython/commit/b8dbb3a7f96718dddb8bed31130f421316213dc5 commit: b8dbb3a7f96718dddb8bed31130f421316213dc5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T16:30:57+02:00 summary: bpo-45436: Fix tkinter tests with Tcl/Tk 8.6.11+ (GH-29077) (GH-29080) Since v8.6.11, a few configuration options seem to accept an empty value where they did not previously; particularly the `type` of a `Menu` widget, and the `compound` of any ttk widget with a label. Providing an explicit expected error message to `checkEnumParam` bypasses the check of an empty value, which no longer raises `TclError`. (cherry picked from commit 4fe454c6f54b0948af67b53af6c2f35af6377e69) Co-authored-by: Zachary Ware files: M Lib/tkinter/test/test_tkinter/test_widgets.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/tkinter/test/test_tkinter/test_widgets.py b/Lib/tkinter/test/test_tkinter/test_widgets.py index 39334de8cf41c..cc227e579679b 100644 --- a/Lib/tkinter/test/test_tkinter/test_widgets.py +++ b/Lib/tkinter/test/test_tkinter/test_widgets.py @@ -1241,8 +1241,11 @@ def test_configure_title(self): def test_configure_type(self): widget = self.create() - self.checkEnumParam(widget, 'type', - 'normal', 'tearoff', 'menubar') + self.checkEnumParam( + widget, 'type', + 'normal', 'tearoff', 'menubar', + errmsg='bad type "{}": must be normal, tearoff, or menubar', + ) def test_entryconfigure(self): m1 = self.create() diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index a2a4de2e5cc41..904aed0ac2df6 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -169,10 +169,13 @@ def checkImageParam(self, widget, name): errmsg='image "spam" doesn\'t exist') def test_configure_compound(self): + options = 'none text image center top bottom left right'.split() + errmsg = ( + 'bad compound "{}": must be' + f' {", ".join(options[:-1])}, or {options[-1]}' + ) widget = self.create() - self.checkEnumParam(widget, 'compound', - 'none', 'text', 'image', 'center', - 'top', 'bottom', 'left', 'right') + self.checkEnumParam(widget, 'compound', *options, errmsg=errmsg) def test_configure_state(self): widget = self.create() From webhook-mailer at python.org Wed Oct 20 10:36:37 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 14:36:37 -0000 Subject: [Python-checkins] bpo-45229: Make doctest tests discoverable (GH-28986) Message-ID: https://github.com/python/cpython/commit/8d6740f489fca67a44de165d29d9e0ad86285779 commit: 8d6740f489fca67a44de165d29d9e0ad86285779 branch: main author: Serhiy Storchaka committer: ambv date: 2021-10-20T16:36:27+02:00 summary: bpo-45229: Make doctest tests discoverable (GH-28986) files: M Lib/test/test_doctest.py M Lib/test/test_doctest2.py diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 571dc78bf5076..3524a0a797c41 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -3114,20 +3114,11 @@ def test_no_trailing_whitespace_stripping(): patches that contain trailing whitespace. More info on Issue 24746. """ -###################################################################### -## Main -###################################################################### - -def test_main(): - # Check the doctest cases in doctest itself: - ret = support.run_doctest(doctest, verbosity=True) - # Check the doctest cases defined here: - from test import test_doctest - support.run_doctest(test_doctest, verbosity=True) - - # Run unittests - support.run_unittest(__name__) +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(doctest)) + tests.addTest(doctest.DocTestSuite()) + return tests def test_coverage(coverdir): @@ -3140,8 +3131,9 @@ def test_coverage(coverdir): r.write_results(show_missing=True, summary=True, coverdir=coverdir) + if __name__ == '__main__': if '-c' in sys.argv: test_coverage('/tmp/doctest.cover') else: - test_main() + unittest.main() diff --git a/Lib/test/test_doctest2.py b/Lib/test/test_doctest2.py index 347a143641071..ab8a0696736e2 100644 --- a/Lib/test/test_doctest2.py +++ b/Lib/test/test_doctest2.py @@ -13,7 +13,6 @@ import sys import unittest -from test import support if sys.flags.optimize >= 2: raise unittest.SkipTest("Cannot test docstrings with -O2") @@ -107,17 +106,21 @@ def clsm(cls, val): """ return val -def test_main(): - from test import test_doctest2 - EXPECTED = 19 - f, t = support.run_doctest(test_doctest2) - if t != EXPECTED: - raise support.TestFailed("expected %d tests to run, not %d" % - (EXPECTED, t)) + +class Test(unittest.TestCase): + def test_testmod(self): + import doctest, sys + EXPECTED = 19 + f, t = doctest.testmod(sys.modules[__name__]) + if f: + self.fail("%d of %d doctests failed" % (f, t)) + if t != EXPECTED: + self.fail("expected %d tests to run, not %d" % (EXPECTED, t)) + # Pollute the namespace with a bunch of imported functions and classes, # to make sure they don't get tested. from doctest import * if __name__ == '__main__': - test_main() + unittest.main() From webhook-mailer at python.org Wed Oct 20 11:08:44 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 20 Oct 2021 15:08:44 -0000 Subject: [Python-checkins] bpo-44525: Specialize simple Python calls. (GH-29033) Message-ID: https://github.com/python/cpython/commit/8863a0fcc5f04ab7c3428e713917831f9b1deb18 commit: 8863a0fcc5f04ab7c3428e713917831f9b1deb18 branch: main author: Mark Shannon committer: markshannon date: 2021-10-20T16:08:28+01:00 summary: bpo-44525: Specialize simple Python calls. (GH-29033) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-20-11-57-31.bpo-44525.veL4lJ.rst M Include/internal/pycore_code.h M Include/opcode.h M Lib/opcode.py M Python/ceval.c M Python/opcode_targets.h M Python/specialize.c diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index d464f3d2a8131..482bd7eb6ae70 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -35,6 +35,12 @@ typedef struct { PyObject *obj; } _PyObjectCache; +typedef struct { + uint32_t func_version; + uint16_t defaults_start; + uint16_t defaults_len; +} _PyCallCache; + /* Add specialized versions of entries to this union. * * Do not break the invariant: sizeof(SpecializedCacheEntry) == 8 @@ -51,6 +57,7 @@ typedef union { _PyAttrCache attr; _PyLoadGlobalCache load_global; _PyObjectCache obj; + _PyCallCache call; } SpecializedCacheEntry; #define INSTRUCTIONS_PER_ENTRY (sizeof(SpecializedCacheEntry)/sizeof(_Py_CODEUNIT)) diff --git a/Include/opcode.h b/Include/opcode.h index 22d968ee0d4c7..f8c02b840e052 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -153,29 +153,30 @@ extern "C" { #define CALL_FUNCTION_BUILTIN_FAST 45 #define CALL_FUNCTION_LEN 46 #define CALL_FUNCTION_ISINSTANCE 47 -#define JUMP_ABSOLUTE_QUICK 48 -#define LOAD_ATTR_ADAPTIVE 58 -#define LOAD_ATTR_INSTANCE_VALUE 80 -#define LOAD_ATTR_WITH_HINT 81 -#define LOAD_ATTR_SLOT 87 -#define LOAD_ATTR_MODULE 88 -#define LOAD_GLOBAL_ADAPTIVE 120 -#define LOAD_GLOBAL_MODULE 122 -#define LOAD_GLOBAL_BUILTIN 123 -#define LOAD_METHOD_ADAPTIVE 127 -#define LOAD_METHOD_CACHED 128 -#define LOAD_METHOD_CLASS 134 -#define LOAD_METHOD_MODULE 140 -#define LOAD_METHOD_NO_DICT 143 -#define STORE_ATTR_ADAPTIVE 149 -#define STORE_ATTR_INSTANCE_VALUE 150 -#define STORE_ATTR_SLOT 151 -#define STORE_ATTR_WITH_HINT 153 -#define LOAD_FAST__LOAD_FAST 154 -#define STORE_FAST__LOAD_FAST 158 -#define LOAD_FAST__LOAD_CONST 159 -#define LOAD_CONST__LOAD_FAST 167 -#define STORE_FAST__STORE_FAST 168 +#define CALL_FUNCTION_PY_SIMPLE 48 +#define JUMP_ABSOLUTE_QUICK 58 +#define LOAD_ATTR_ADAPTIVE 80 +#define LOAD_ATTR_INSTANCE_VALUE 81 +#define LOAD_ATTR_WITH_HINT 87 +#define LOAD_ATTR_SLOT 88 +#define LOAD_ATTR_MODULE 120 +#define LOAD_GLOBAL_ADAPTIVE 122 +#define LOAD_GLOBAL_MODULE 123 +#define LOAD_GLOBAL_BUILTIN 127 +#define LOAD_METHOD_ADAPTIVE 128 +#define LOAD_METHOD_CACHED 134 +#define LOAD_METHOD_CLASS 140 +#define LOAD_METHOD_MODULE 143 +#define LOAD_METHOD_NO_DICT 149 +#define STORE_ATTR_ADAPTIVE 150 +#define STORE_ATTR_INSTANCE_VALUE 151 +#define STORE_ATTR_SLOT 153 +#define STORE_ATTR_WITH_HINT 154 +#define LOAD_FAST__LOAD_FAST 158 +#define STORE_FAST__LOAD_FAST 159 +#define LOAD_FAST__LOAD_CONST 167 +#define LOAD_CONST__LOAD_FAST 168 +#define STORE_FAST__STORE_FAST 169 #define DO_TRACING 255 #ifdef NEED_OPCODE_JUMP_TABLES static uint32_t _PyOpcode_RelativeJump[8] = { diff --git a/Lib/opcode.py b/Lib/opcode.py index fe6066fc517a6..5377ec32bf153 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -237,6 +237,7 @@ def jabs_op(name, op): "CALL_FUNCTION_BUILTIN_FAST", "CALL_FUNCTION_LEN", "CALL_FUNCTION_ISINSTANCE", + "CALL_FUNCTION_PY_SIMPLE", "JUMP_ABSOLUTE_QUICK", "LOAD_ATTR_ADAPTIVE", "LOAD_ATTR_INSTANCE_VALUE", diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-20-11-57-31.bpo-44525.veL4lJ.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-20-11-57-31.bpo-44525.veL4lJ.rst new file mode 100644 index 0000000000000..6ab1d05603db8 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-20-11-57-31.bpo-44525.veL4lJ.rst @@ -0,0 +1 @@ +Specialize simple calls to Python functions (no starargs, keyowrd dict, or closure) diff --git a/Python/ceval.c b/Python/ceval.c index 76325903149cd..f4186dae8a448 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4720,9 +4720,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(CALL_FUNCTION_ADAPTIVE) { SpecializedCacheEntry *cache = GET_CACHE(); + nargs = cache->adaptive.original_oparg; if (cache->adaptive.counter == 0) { next_instr--; - int nargs = cache->adaptive.original_oparg; if (_Py_Specialize_CallFunction( PEEK(nargs + 1), next_instr, nargs, cache, BUILTINS()) < 0) { goto error; @@ -4732,9 +4732,48 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr else { STAT_INC(CALL_FUNCTION, deferred); cache->adaptive.counter--; - oparg = cache->adaptive.original_oparg; - JUMP_TO_INSTRUCTION(CALL_FUNCTION); + oparg = nargs; + kwnames = NULL; + postcall_shrink = 1; + goto call_function; + } + } + + TARGET(CALL_FUNCTION_PY_SIMPLE) { + SpecializedCacheEntry *caches = GET_CACHE(); + _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + int argcount = cache0->original_oparg; + _PyCallCache *cache1 = &caches[-1].call; + PyObject *callable = PEEK(argcount+1); + DEOPT_IF(!PyFunction_Check(callable), CALL_FUNCTION); + PyFunctionObject *func = (PyFunctionObject *)callable; + DEOPT_IF(func->func_version != cache1->func_version, CALL_FUNCTION); + /* PEP 523 */ + DEOPT_IF(tstate->interp->eval_frame != NULL, CALL_FUNCTION); + STAT_INC(CALL_FUNCTION, hit); + record_cache_hit(cache0); + InterpreterFrame *new_frame = _PyThreadState_PushFrame( + tstate, PyFunction_AS_FRAME_CONSTRUCTOR(func), NULL); + if (new_frame == NULL) { + goto error; + } + STACK_SHRINK(argcount); + for (int i = 0; i < argcount; i++) { + new_frame->localsplus[i] = stack_pointer[i]; + } + int deflen = cache1->defaults_len; + for (int i = 0; i < deflen; i++) { + PyObject *def = PyTuple_GET_ITEM(func->func_defaults, cache1->defaults_start+i); + Py_INCREF(def); + new_frame->localsplus[argcount+i] = def; } + STACK_SHRINK(1); + Py_DECREF(func); + _PyFrame_SetStackPointer(frame, stack_pointer); + new_frame->previous = tstate->frame; + new_frame->depth = frame->depth + 1; + tstate->frame = frame = new_frame; + goto start_frame; } TARGET(CALL_FUNCTION_BUILTIN_O) { diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 4179689e8a866..5c7d3ad544e56 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -47,7 +47,7 @@ static void *opcode_targets[256] = { &&TARGET_CALL_FUNCTION_BUILTIN_FAST, &&TARGET_CALL_FUNCTION_LEN, &&TARGET_CALL_FUNCTION_ISINSTANCE, - &&TARGET_JUMP_ABSOLUTE_QUICK, + &&TARGET_CALL_FUNCTION_PY_SIMPLE, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, @@ -57,7 +57,7 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_ADD, &&TARGET_INPLACE_SUBTRACT, &&TARGET_INPLACE_MULTIPLY, - &&TARGET_LOAD_ATTR_ADAPTIVE, + &&TARGET_JUMP_ABSOLUTE_QUICK, &&TARGET_INPLACE_MODULO, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, @@ -79,15 +79,15 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_AND, &&TARGET_INPLACE_XOR, &&TARGET_INPLACE_OR, + &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_LOAD_ATTR_INSTANCE_VALUE, - &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, &&TARGET_SETUP_ANNOTATIONS, &&TARGET_YIELD_VALUE, + &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LOAD_ATTR_SLOT, - &&TARGET_LOAD_ATTR_MODULE, &&TARGET_POP_EXCEPT, &&TARGET_STORE_NAME, &&TARGET_DELETE_NAME, @@ -119,46 +119,46 @@ static void *opcode_targets[256] = { &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, - &&TARGET_LOAD_GLOBAL_ADAPTIVE, + &&TARGET_LOAD_ATTR_MODULE, &&TARGET_JUMP_IF_NOT_EXC_MATCH, + &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_LOAD_GLOBAL_MODULE, - &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LOAD_FAST, &&TARGET_STORE_FAST, &&TARGET_DELETE_FAST, + &&TARGET_LOAD_GLOBAL_BUILTIN, &&TARGET_LOAD_METHOD_ADAPTIVE, - &&TARGET_LOAD_METHOD_CACHED, &&TARGET_GEN_START, &&TARGET_RAISE_VARARGS, &&TARGET_CALL_FUNCTION, &&TARGET_MAKE_FUNCTION, &&TARGET_BUILD_SLICE, - &&TARGET_LOAD_METHOD_CLASS, + &&TARGET_LOAD_METHOD_CACHED, &&TARGET_MAKE_CELL, &&TARGET_LOAD_CLOSURE, &&TARGET_LOAD_DEREF, &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, - &&TARGET_LOAD_METHOD_MODULE, + &&TARGET_LOAD_METHOD_CLASS, &&TARGET_CALL_FUNCTION_KW, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_LOAD_METHOD_NO_DICT, + &&TARGET_LOAD_METHOD_MODULE, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, + &&TARGET_LOAD_METHOD_NO_DICT, &&TARGET_STORE_ATTR_ADAPTIVE, &&TARGET_STORE_ATTR_INSTANCE_VALUE, - &&TARGET_STORE_ATTR_SLOT, &&TARGET_MATCH_CLASS, + &&TARGET_STORE_ATTR_SLOT, &&TARGET_STORE_ATTR_WITH_HINT, - &&TARGET_LOAD_FAST__LOAD_FAST, &&TARGET_FORMAT_VALUE, &&TARGET_BUILD_CONST_KEY_MAP, &&TARGET_BUILD_STRING, + &&TARGET_LOAD_FAST__LOAD_FAST, &&TARGET_STORE_FAST__LOAD_FAST, - &&TARGET_LOAD_FAST__LOAD_CONST, &&TARGET_LOAD_METHOD, &&TARGET_CALL_METHOD, &&TARGET_LIST_EXTEND, @@ -166,6 +166,7 @@ static void *opcode_targets[256] = { &&TARGET_DICT_MERGE, &&TARGET_DICT_UPDATE, &&TARGET_CALL_METHOD_KW, + &&TARGET_LOAD_FAST__LOAD_CONST, &&TARGET_LOAD_CONST__LOAD_FAST, &&TARGET_STORE_FAST__STORE_FAST, &&_unknown_opcode, @@ -253,6 +254,5 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_DO_TRACING }; diff --git a/Python/specialize.c b/Python/specialize.c index ee573d29a474e..5cc7082a35a21 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -249,7 +249,7 @@ static uint8_t cache_requirements[256] = { [BINARY_ADD] = 0, [BINARY_MULTIPLY] = 0, [BINARY_SUBSCR] = 0, - [CALL_FUNCTION] = 2, /* _PyAdaptiveEntry and _PyObjectCache */ + [CALL_FUNCTION] = 2, /* _PyAdaptiveEntry and _PyObjectCache/_PyCallCache */ [STORE_ATTR] = 2, /* _PyAdaptiveEntry and _PyAttrCache */ }; @@ -461,15 +461,20 @@ _Py_Quicken(PyCodeObject *code) { #define SPEC_FAIL_NON_FUNCTION_SCOPE 11 #define SPEC_FAIL_DIFFERENT_TYPES 12 -/* Call function */ +/* Calls */ +#define SPEC_FAIL_GENERATOR 7 +#define SPEC_FAIL_COMPLEX_PARAMETERS 8 +#define SPEC_FAIL_WRONG_NUMBER_ARGUMENTS 9 +#define SPEC_FAIL_CO_NOT_OPTIMIZED 10 +/* SPEC_FAIL_METHOD defined as 11 above */ +#define SPEC_FAIL_FREE_VARS 12 +#define SPEC_FAIL_PYCFUNCTION 13 +#define SPEC_FAIL_PYCFUNCTION_WITH_KEYWORDS 14 +#define SPEC_FAIL_PYCFUNCTION_FAST_WITH_KEYWORDS 15 +#define SPEC_FAIL_PYCFUNCTION_NOARGS 16 +#define SPEC_FAIL_BAD_CALL_FLAGS 17 +#define SPEC_FAIL_CLASS 18 -#define SPEC_FAIL_PYCFUNCTION 10 -#define SPEC_FAIL_PYCFUNCTION_WITH_KEYWORDS 13 -#define SPEC_FAIL_PYCFUNCTION_FAST_WITH_KEYWORDS 14 -#define SPEC_FAIL_PYCFUNCTION_NOARGS 15 -#define SPEC_FAIL_BAD_CALL_FLAGS 16 -#define SPEC_FAIL_PYTHON_FUNCTION 17 -#define SPEC_FAIL_IMMUTABLE_CLASS 18 static int specialize_module_load_attr( @@ -1236,6 +1241,69 @@ _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *ins return 0; } +static int +specialize_class_call( + PyObject *callable, _Py_CODEUNIT *instr, + int nargs, SpecializedCacheEntry *cache) +{ + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_CLASS); + return -1; +} + +static int +specialize_py_call( + PyFunctionObject *func, _Py_CODEUNIT *instr, + int nargs, SpecializedCacheEntry *cache) +{ + _PyCallCache *cache1 = &cache[-1].call; + /* Exclude generator or coroutines for now */ + PyCodeObject *code = (PyCodeObject *)func->func_code; + int flags = code->co_flags; + if (flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_GENERATOR); + return -1; + } + if ((flags & (CO_VARKEYWORDS | CO_VARARGS)) || code->co_kwonlyargcount) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_COMPLEX_PARAMETERS); + return -1; + } + if ((flags & CO_OPTIMIZED) == 0) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_CO_NOT_OPTIMIZED); + return -1; + } + if (code->co_nfreevars) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_FREE_VARS); + return -1; + } + int argcount = code->co_argcount; + int defcount = func->func_defaults == NULL ? 0 : (int)PyTuple_GET_SIZE(func->func_defaults); + assert(defcount <= argcount); + int min_args = argcount-defcount; + if (nargs > argcount || nargs < min_args) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_WRONG_NUMBER_ARGUMENTS); + return -1; + } + assert(nargs <= argcount && nargs >= min_args); + int defstart = nargs - min_args; + int deflen = argcount - nargs; + assert(defstart >= 0 && deflen >= 0); + assert(deflen == 0 || func->func_defaults != NULL); + if (defstart > 0xffff || deflen > 0xffff) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_OUT_OF_RANGE); + return -1; + } + int version = _PyFunction_GetVersionForCurrentState(func); + if (version == 0) { + SPECIALIZATION_FAIL(CALL_FUNCTION, SPEC_FAIL_OUT_OF_VERSIONS); + return -1; + } + cache1->func_version = version; + cache1->defaults_start = defstart; + cache1->defaults_len = deflen; + *instr = _Py_MAKECODEUNIT(CALL_FUNCTION_PY_SIMPLE, _Py_OPARG(*instr)); + return 0; +} + #if COLLECT_SPECIALIZATION_STATS_DETAILED static int builtin_call_fail_kind(int ml_flags) @@ -1315,11 +1383,7 @@ specialize_c_call(PyObject *callable, _Py_CODEUNIT *instr, int nargs, static int call_fail_kind(PyObject *callable) { - if (PyFunction_Check(callable)) { - return SPEC_FAIL_PYTHON_FUNCTION; - } - // new-style bound methods - else if (PyInstanceMethod_Check(callable)) { + if (PyInstanceMethod_Check(callable)) { return SPEC_FAIL_METHOD; } else if (PyMethod_Check(callable)) { @@ -1330,17 +1394,14 @@ call_fail_kind(PyObject *callable) return SPEC_FAIL_METHOD; } else if (PyType_Check(callable)) { - PyTypeObject *type = Py_TYPE(callable); - return PyType_HasFeature(type, Py_TPFLAGS_IMMUTABLETYPE) ? - SPEC_FAIL_IMMUTABLE_CLASS : SPEC_FAIL_MUTABLE_CLASS; + return SPEC_FAIL_CLASS; } return SPEC_FAIL_OTHER; } #endif /* TODO: - - Specialize calling types. - - Specialize python function calls. + - Specialize calling classes. */ int _Py_Specialize_CallFunction( @@ -1352,9 +1413,15 @@ _Py_Specialize_CallFunction( if (PyCFunction_CheckExact(callable)) { fail = specialize_c_call(callable, instr, nargs, cache, builtins); } + else if (PyFunction_Check(callable)) { + fail = specialize_py_call((PyFunctionObject *)callable, instr, nargs, cache); + } + else if (PyType_Check(callable)) { + fail = specialize_class_call(callable, instr, nargs, cache); + } else { SPECIALIZATION_FAIL(CALL_FUNCTION, call_fail_kind(callable)); - fail = 1; + fail = -1; } _PyAdaptiveEntry *cache0 = &cache->adaptive; if (fail) { From webhook-mailer at python.org Wed Oct 20 11:09:05 2021 From: webhook-mailer at python.org (warsaw) Date: Wed, 20 Oct 2021 15:09:05 -0000 Subject: [Python-checkins] Add a comment about how to fix bogus test_host_resolution_bad_address failures (#29085) Message-ID: https://github.com/python/cpython/commit/6ef4507c74106884e6c8847f08d004511c4f1197 commit: 6ef4507c74106884e6c8847f08d004511c4f1197 branch: main author: Barry Warsaw committer: warsaw date: 2021-10-20T08:08:57-07:00 summary: Add a comment about how to fix bogus test_host_resolution_bad_address failures (#29085) files: M Lib/test/test_socket.py diff --git a/Lib/test/test_socket.py b/Lib/test/test_socket.py index eeb8e8c98a149..394d2942483fb 100755 --- a/Lib/test/test_socket.py +++ b/Lib/test/test_socket.py @@ -1017,8 +1017,10 @@ def test_host_resolution(self): def test_host_resolution_bad_address(self): # These are all malformed IP addresses and expected not to resolve to - # any result. But some ISPs, e.g. AWS, may successfully resolve these - # IPs. + # any result. But some ISPs, e.g. AWS and AT&T, may successfully + # resolve these IPs. In particular, AT&T's DNS Error Assist service + # will break this test. See https://bugs.python.org/issue42092 for a + # workaround. explanation = ( "resolving an invalid IP address did not raise OSError; " "can be caused by a broken DNS server" From webhook-mailer at python.org Wed Oct 20 11:16:04 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Wed, 20 Oct 2021 15:16:04 -0000 Subject: [Python-checkins] Add PEPs 593 & 647 to list of PEPs at top of typing docs (GH-29097) Message-ID: https://github.com/python/cpython/commit/d9e1dae35ac20acfeb2509b0dea4c3943693e79d commit: d9e1dae35ac20acfeb2509b0dea4c3943693e79d branch: main author: Alex Waygood committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-20T23:15:59+08:00 summary: Add PEPs 593 & 647 to list of PEPs at top of typing docs (GH-29097) files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index bc6130ace4a4b..e5e7941833beb 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -19,7 +19,7 @@ This module provides runtime support for type hints as specified by :pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`, -:pep:`612` and :pep:`613`. +:pep:`593`, :pep:`612`, :pep:`613` and :pep:`647`. The most fundamental support consists of the types :data:`Any`, :data:`Union`, :data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For full specification please see :pep:`484`. For From webhook-mailer at python.org Wed Oct 20 11:18:43 2021 From: webhook-mailer at python.org (tiran) Date: Wed, 20 Oct 2021 15:18:43 -0000 Subject: [Python-checkins] bpo-45536: Check OpenSSL APIs in configure (GH-29088) Message-ID: https://github.com/python/cpython/commit/81520fe677d15cc7f9af5140bc5f9eca8409ad90 commit: 81520fe677d15cc7f9af5140bc5f9eca8409ad90 branch: main author: Christian Heimes committer: tiran date: 2021-10-20T17:18:34+02:00 summary: bpo-45536: Check OpenSSL APIs in configure (GH-29088) files: A Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst b/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst new file mode 100644 index 0000000000000..e560b71ede2d2 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst @@ -0,0 +1,3 @@ +The ``configure`` script now checks whether OpenSSL headers and libraries +provide required APIs. Most common APIs are verified. The check detects +outdated or missing OpenSSL. Failures do not stop configure. diff --git a/configure b/configure index 81ee4282d9412..ec7c72c7f3276 100755 --- a/configure +++ b/configure @@ -17949,6 +17949,66 @@ esac $as_echo "$OPENSSL_RPATH" >&6; } +# check if OpenSSL libraries work as expected +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether OpenSSL provides required APIs" >&5 +$as_echo_n "checking whether OpenSSL provides required APIs... " >&6; } +save_LIBS="$LIBS" +save_CFLAGS="$CFLAGS" +save_LDFLAGS="$LDFLAGS" +LIBS="$LIBS $OPENSSL_LIBS" +CFLAGS="$CFLAGS_NODIST $OPENSSL_INCLUDES" +LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS" + +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +#include +#include +#include + +#if OPENSSL_VERSION_NUMBER < 0x10101000L +#error "OpenSSL >= 1.1.1 is required" +#endif + +static void keylog_cb(const SSL *ssl, const char *line) {} + +int +main () +{ + +/* SSL APIs */ +SSL_CTX *ctx = SSL_CTX_new(TLS_client_method()); +SSL_CTX_set_keylog_callback(ctx, keylog_cb); +SSL *ssl = SSL_new(ctx); +X509_VERIFY_PARAM *param = SSL_get0_param(ssl); +X509_VERIFY_PARAM_set1_host(param, "python.org", 0); +SSL_free(ssl); +SSL_CTX_free(ctx); + +/* hashlib APIs */ +OBJ_nid2sn(NID_md5); +OBJ_nid2sn(NID_sha1); +OBJ_nid2sn(NID_sha3_512); +OBJ_nid2sn(NID_blake2b512); +EVP_PBE_scrypt(NULL, 0, NULL, 0, 2, 8, 1, 0, NULL, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS="$save_LIBS" +CFLAGS="$save_CFLAGS" +LDFLAGS="$save_LDFLAGS" + # ssl module default cipher suite string diff --git a/configure.ac b/configure.ac index ab3fc2839d4f8..c0259524756f2 100644 --- a/configure.ac +++ b/configure.ac @@ -5888,6 +5888,48 @@ AS_CASE($with_openssl_rpath, AC_MSG_RESULT($OPENSSL_RPATH) AC_SUBST([OPENSSL_RPATH]) +# check if OpenSSL libraries work as expected +AC_MSG_CHECKING(whether OpenSSL provides required APIs) +save_LIBS="$LIBS" +save_CFLAGS="$CFLAGS" +save_LDFLAGS="$LDFLAGS" +LIBS="$LIBS $OPENSSL_LIBS" +CFLAGS="$CFLAGS_NODIST $OPENSSL_INCLUDES" +LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS" + +AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#include +#include +#include + +#if OPENSSL_VERSION_NUMBER < 0x10101000L +#error "OpenSSL >= 1.1.1 is required" +#endif + +static void keylog_cb(const SSL *ssl, const char *line) {} +]], [[ +/* SSL APIs */ +SSL_CTX *ctx = SSL_CTX_new(TLS_client_method()); +SSL_CTX_set_keylog_callback(ctx, keylog_cb); +SSL *ssl = SSL_new(ctx); +X509_VERIFY_PARAM *param = SSL_get0_param(ssl); +X509_VERIFY_PARAM_set1_host(param, "python.org", 0); +SSL_free(ssl); +SSL_CTX_free(ctx); + +/* hashlib APIs */ +OBJ_nid2sn(NID_md5); +OBJ_nid2sn(NID_sha1); +OBJ_nid2sn(NID_sha3_512); +OBJ_nid2sn(NID_blake2b512); +EVP_PBE_scrypt(NULL, 0, NULL, 0, 2, 8, 1, 0, NULL, 0); +]])], + [AC_MSG_RESULT(yes)], + [AC_MSG_RESULT(no)]) +LIBS="$save_LIBS" +CFLAGS="$save_CFLAGS" +LDFLAGS="$save_LDFLAGS" + # ssl module default cipher suite string AH_TEMPLATE(PY_SSL_DEFAULT_CIPHERS, [Default cipher suites list for ssl module. From webhook-mailer at python.org Wed Oct 20 11:36:42 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 15:36:42 -0000 Subject: [Python-checkins] Add PEPs 593 & 647 to list of PEPs at top of typing docs (GH-29097) Message-ID: https://github.com/python/cpython/commit/0d0312e1ac4a92d8cfb71c4c66c66eaf7270262f commit: 0d0312e1ac4a92d8cfb71c4c66c66eaf7270262f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T08:36:34-07:00 summary: Add PEPs 593 & 647 to list of PEPs at top of typing docs (GH-29097) (cherry picked from commit d9e1dae35ac20acfeb2509b0dea4c3943693e79d) Co-authored-by: Alex Waygood files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 13760c19214ee..c402c73b487ea 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -19,7 +19,7 @@ This module provides runtime support for type hints as specified by :pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`, -:pep:`612` and :pep:`613`. +:pep:`593`, :pep:`612`, :pep:`613` and :pep:`647`. The most fundamental support consists of the types :data:`Any`, :data:`Union`, :data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For full specification please see :pep:`484`. For From webhook-mailer at python.org Wed Oct 20 11:46:44 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 15:46:44 -0000 Subject: [Python-checkins] bpo-45536: Check OpenSSL APIs in configure (GH-29088) Message-ID: https://github.com/python/cpython/commit/5537b9f10510735447bea81079ac586f46decf20 commit: 5537b9f10510735447bea81079ac586f46decf20 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T08:46:32-07:00 summary: bpo-45536: Check OpenSSL APIs in configure (GH-29088) (cherry picked from commit 81520fe677d15cc7f9af5140bc5f9eca8409ad90) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst M configure M configure.ac diff --git a/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst b/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst new file mode 100644 index 0000000000000..e560b71ede2d2 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-12-42-39.bpo-45536.oQNYHB.rst @@ -0,0 +1,3 @@ +The ``configure`` script now checks whether OpenSSL headers and libraries +provide required APIs. Most common APIs are verified. The check detects +outdated or missing OpenSSL. Failures do not stop configure. diff --git a/configure b/configure index a6e0f823b453a..e79425d5b9d8e 100755 --- a/configure +++ b/configure @@ -17778,6 +17778,66 @@ esac $as_echo "$OPENSSL_RPATH" >&6; } +# check if OpenSSL libraries work as expected +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether OpenSSL provides required APIs" >&5 +$as_echo_n "checking whether OpenSSL provides required APIs... " >&6; } +save_LIBS="$LIBS" +save_CFLAGS="$CFLAGS" +save_LDFLAGS="$LDFLAGS" +LIBS="$LIBS $OPENSSL_LIBS" +CFLAGS="$CFLAGS_NODIST $OPENSSL_INCLUDES" +LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS" + +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +#include +#include +#include + +#if OPENSSL_VERSION_NUMBER < 0x10101000L +#error "OpenSSL >= 1.1.1 is required" +#endif + +static void keylog_cb(const SSL *ssl, const char *line) {} + +int +main () +{ + +/* SSL APIs */ +SSL_CTX *ctx = SSL_CTX_new(TLS_client_method()); +SSL_CTX_set_keylog_callback(ctx, keylog_cb); +SSL *ssl = SSL_new(ctx); +X509_VERIFY_PARAM *param = SSL_get0_param(ssl); +X509_VERIFY_PARAM_set1_host(param, "python.org", 0); +SSL_free(ssl); +SSL_CTX_free(ctx); + +/* hashlib APIs */ +OBJ_nid2sn(NID_md5); +OBJ_nid2sn(NID_sha1); +OBJ_nid2sn(NID_sha3_512); +OBJ_nid2sn(NID_blake2b512); +EVP_PBE_scrypt(NULL, 0, NULL, 0, 2, 8, 1, 0, NULL, 0); + + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } +else + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS="$save_LIBS" +CFLAGS="$save_CFLAGS" +LDFLAGS="$save_LDFLAGS" + # ssl module default cipher suite string diff --git a/configure.ac b/configure.ac index c4abac6701152..4c2ac5915a1cf 100644 --- a/configure.ac +++ b/configure.ac @@ -5831,6 +5831,48 @@ AS_CASE($with_openssl_rpath, AC_MSG_RESULT($OPENSSL_RPATH) AC_SUBST([OPENSSL_RPATH]) +# check if OpenSSL libraries work as expected +AC_MSG_CHECKING(whether OpenSSL provides required APIs) +save_LIBS="$LIBS" +save_CFLAGS="$CFLAGS" +save_LDFLAGS="$LDFLAGS" +LIBS="$LIBS $OPENSSL_LIBS" +CFLAGS="$CFLAGS_NODIST $OPENSSL_INCLUDES" +LDFLAGS="$LDFLAGS $OPENSSL_LDFLAGS" + +AC_LINK_IFELSE([AC_LANG_PROGRAM([[ +#include +#include +#include + +#if OPENSSL_VERSION_NUMBER < 0x10101000L +#error "OpenSSL >= 1.1.1 is required" +#endif + +static void keylog_cb(const SSL *ssl, const char *line) {} +]], [[ +/* SSL APIs */ +SSL_CTX *ctx = SSL_CTX_new(TLS_client_method()); +SSL_CTX_set_keylog_callback(ctx, keylog_cb); +SSL *ssl = SSL_new(ctx); +X509_VERIFY_PARAM *param = SSL_get0_param(ssl); +X509_VERIFY_PARAM_set1_host(param, "python.org", 0); +SSL_free(ssl); +SSL_CTX_free(ctx); + +/* hashlib APIs */ +OBJ_nid2sn(NID_md5); +OBJ_nid2sn(NID_sha1); +OBJ_nid2sn(NID_sha3_512); +OBJ_nid2sn(NID_blake2b512); +EVP_PBE_scrypt(NULL, 0, NULL, 0, 2, 8, 1, 0, NULL, 0); +]])], + [AC_MSG_RESULT(yes)], + [AC_MSG_RESULT(no)]) +LIBS="$save_LIBS" +CFLAGS="$save_CFLAGS" +LDFLAGS="$save_LDFLAGS" + # ssl module default cipher suite string AH_TEMPLATE(PY_SSL_DEFAULT_CIPHERS, [Default cipher suites list for ssl module. From webhook-mailer at python.org Wed Oct 20 12:18:01 2021 From: webhook-mailer at python.org (gpshead) Date: Wed, 20 Oct 2021 16:18:01 -0000 Subject: [Python-checkins] Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) Message-ID: https://github.com/python/cpython/commit/1dfac27dffbe771f9d88bd1726f7362ce0341437 commit: 1dfac27dffbe771f9d88bd1726f7362ce0341437 branch: main author: Gregory P. Smith committer: gpshead date: 2021-10-20T09:17:52-07:00 summary: Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) files: M Lib/asyncio/unix_events.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 4cef914b9fb9e..c88b818de62a6 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1379,7 +1379,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classes requires it + # abstract base classes require it. return True def attach_loop(self, loop): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 904aed0ac2df6..935be3d7f1b5a 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -971,7 +971,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets read at its previous position + # verify that the tab gets re-added at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) From webhook-mailer at python.org Wed Oct 20 12:42:43 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 16:42:43 -0000 Subject: [Python-checkins] Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) Message-ID: https://github.com/python/cpython/commit/d6afe3be0106818454e1aac07bebe6d5e75bb38d commit: d6afe3be0106818454e1aac07bebe6d5e75bb38d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T09:42:39-07:00 summary: Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) (cherry picked from commit 1dfac27dffbe771f9d88bd1726f7362ce0341437) Co-authored-by: Gregory P. Smith files: M Lib/asyncio/unix_events.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 4cef914b9fb9e..c88b818de62a6 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1379,7 +1379,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classes requires it + # abstract base classes require it. return True def attach_loop(self, loop): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 904aed0ac2df6..935be3d7f1b5a 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -971,7 +971,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets read at its previous position + # verify that the tab gets re-added at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) From webhook-mailer at python.org Wed Oct 20 12:50:36 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 16:50:36 -0000 Subject: [Python-checkins] Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) Message-ID: https://github.com/python/cpython/commit/50e8b2ff0273e81829ed3fa4ab9ef9898fd2b891 commit: 50e8b2ff0273e81829ed3fa4ab9ef9898fd2b891 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T09:50:27-07:00 summary: Cleanup a couple of comments left on PR 28775 post-merge. (GH-29079) (cherry picked from commit 1dfac27dffbe771f9d88bd1726f7362ce0341437) Co-authored-by: Gregory P. Smith files: M Lib/asyncio/unix_events.py M Lib/tkinter/test/test_ttk/test_widgets.py diff --git a/Lib/asyncio/unix_events.py b/Lib/asyncio/unix_events.py index 9ab5dba272fd1..eecbc101ee1b9 100644 --- a/Lib/asyncio/unix_events.py +++ b/Lib/asyncio/unix_events.py @@ -1383,7 +1383,7 @@ def add_child_handler(self, pid, callback, *args): def remove_child_handler(self, pid): # asyncio never calls remove_child_handler() !!! # The method is no-op but is implemented because - # abstract base classes requires it + # abstract base classes require it. return True def attach_loop(self, loop): diff --git a/Lib/tkinter/test/test_ttk/test_widgets.py b/Lib/tkinter/test/test_ttk/test_widgets.py index 904aed0ac2df6..935be3d7f1b5a 100644 --- a/Lib/tkinter/test/test_ttk/test_widgets.py +++ b/Lib/tkinter/test/test_ttk/test_widgets.py @@ -971,7 +971,7 @@ def test_add_and_hidden(self): tabs = self.nb.tabs() curr = self.nb.index('current') - # verify that the tab gets read at its previous position + # verify that the tab gets re-added at its previous position child2_index = self.nb.index(self.child2) self.nb.hide(self.child2) self.nb.add(self.child2) From webhook-mailer at python.org Wed Oct 20 12:51:19 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 16:51:19 -0000 Subject: [Python-checkins] [3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (#29071) Message-ID: https://github.com/python/cpython/commit/88f4ec88e282bf861f0af2d237e9fe28fbc8deac commit: 88f4ec88e282bf861f0af2d237e9fe28fbc8deac branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-20T18:51:13+02:00 summary: [3.9] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (#29071) There are two errors that this commit fixes: * The parser was not correctly computing the offset and the string source for E_LINECONT errors due to the incorrect usage of strtok(). * The parser was not correctly unwinding the call stack when a tokenizer exception happened in rules involving optionals ('?', [...]) as we always make them return valid results by using the comma operator. We need to check first if we don't have an error before continuing.. (cherry picked from commit a106343f632a99c8ebb0136fa140cf189b4a6a57) Co-authored-by: Pablo Galindo Salgado NOTE: unlike the cherry-picked original, this commit points at a crazy location due to a bug in the tokenizer that required a big refactor in 3.10 to fix. We are leaving as-is for 3.9. files: A Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst M Lib/test/test_exceptions.py M Parser/pegen/parse.c M Parser/pegen/pegen.c M Tools/peg_generator/pegen/c_generator.py diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 70e438d2ee943..4a7ca60b39bc5 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -193,6 +193,10 @@ def check(self, src, lineno, offset, encoding='utf-8'): line = src.split('\n')[lineno-1] self.assertIn(line, cm.exception.text) + def test_error_offset_continuation_characters(self): + check = self.check + check('"\\\n"(1 for c in I,\\\n\\', 3, 22) + def testSyntaxErrorOffset(self): check = self.check check('def fact(x):\n\treturn x!\n', 2, 10) diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst new file mode 100644 index 0000000000000..97e29813ab266 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-16-17-27-48.bpo-45494.vMt1g4.rst @@ -0,0 +1,2 @@ +Fix parser crash when reporting errors involving invalid continuation +characters. Patch by Pablo Galindo. diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c index dc388ce3dbd9e..611257d67de42 100644 --- a/Parser/pegen/parse.c +++ b/Parser/pegen/parse.c @@ -728,7 +728,7 @@ file_rule(Parser *p) void *a; Token * endmarker_var; if ( - (a = statements_rule(p), 1) // statements? + (a = statements_rule(p), !p->error_indicator) // statements? && (endmarker_var = _PyPegen_expect_token(p, ENDMARKER)) // token='ENDMARKER' ) @@ -867,7 +867,7 @@ func_type_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = type_expressions_rule(p), 1) // type_expressions? + (a = type_expressions_rule(p), !p->error_indicator) // type_expressions? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -1459,7 +1459,7 @@ simple_stmt_rule(Parser *p) if ( (a = _gather_12_rule(p)) // ';'.small_stmt+ && - (_opt_var = _PyPegen_expect_token(p, 13), 1) // ';'? + (_opt_var = _PyPegen_expect_token(p, 13), !p->error_indicator) // ';'? && (newline_var = _PyPegen_expect_token(p, NEWLINE)) // token='NEWLINE' ) @@ -2059,7 +2059,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_19_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_19_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ':' expression ['=' annotated_rhs]")); @@ -2101,7 +2101,7 @@ assignment_rule(Parser *p) && (b = expression_rule(p)) // expression && - (c = _tmp_21_rule(p), 1) // ['=' annotated_rhs] + (c = _tmp_21_rule(p), !p->error_indicator) // ['=' annotated_rhs] ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "('(' single_target ')' | single_subscript_attribute_target) ':' expression ['=' annotated_rhs]")); @@ -2142,7 +2142,7 @@ assignment_rule(Parser *p) && _PyPegen_lookahead_with_int(0, _PyPegen_expect_token, p, 22) // token='=' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ assignment[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "((star_targets '='))+ (yield_expr | star_expressions) !'=' TYPE_COMMENT?")); @@ -2797,7 +2797,7 @@ assert_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_29_rule(p), 1) // [',' expression] + (b = _tmp_29_rule(p), !p->error_indicator) // [',' expression] ) { D(fprintf(stderr, "%*c+ assert_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'assert' expression [',' expression]")); @@ -3174,7 +3174,7 @@ import_from_targets_rule(Parser *p) && (a = import_from_as_names_rule(p)) // import_from_as_names && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -3325,7 +3325,7 @@ import_from_as_name_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_35_rule(p), 1) // ['as' NAME] + (b = _tmp_35_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ import_from_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME ['as' NAME]")); @@ -3410,7 +3410,7 @@ dotted_as_name_rule(Parser *p) if ( (a = dotted_name_rule(p)) // dotted_name && - (b = _tmp_38_rule(p), 1) // ['as' NAME] + (b = _tmp_38_rule(p), !p->error_indicator) // ['as' NAME] ) { D(fprintf(stderr, "%*c+ dotted_as_name[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "dotted_name ['as' NAME]")); @@ -3617,7 +3617,7 @@ if_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ if_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'if' named_expression ':' block else_block?")); @@ -3735,7 +3735,7 @@ elif_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ elif_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'elif' named_expression ':' block else_block?")); @@ -3853,7 +3853,7 @@ while_stmt_rule(Parser *p) && (b = block_rule(p)) // block && - (c = else_block_rule(p), 1) // else_block? + (c = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ while_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'while' named_expression ':' block else_block?")); @@ -3935,11 +3935,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'for' star_targets 'in' ~ star_expressions ':' TYPE_COMMENT? block else_block?")); @@ -3999,11 +3999,11 @@ for_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? ) { D(fprintf(stderr, "%*c+ for_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "ASYNC 'for' star_targets 'in' ~ star_expressions ':' TYPE_COMMENT? block else_block?")); @@ -4102,7 +4102,7 @@ with_stmt_rule(Parser *p) && (a = _gather_39_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -4151,7 +4151,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -4202,7 +4202,7 @@ with_stmt_rule(Parser *p) && (a = _gather_43_rule(p)) // ','.with_item+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -4254,7 +4254,7 @@ with_stmt_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && (b = block_rule(p)) // block ) @@ -4467,9 +4467,9 @@ try_stmt_rule(Parser *p) && (ex = _loop1_48_rule(p)) // except_block+ && - (el = else_block_rule(p), 1) // else_block? + (el = else_block_rule(p), !p->error_indicator) // else_block? && - (f = finally_block_rule(p), 1) // finally_block? + (f = finally_block_rule(p), !p->error_indicator) // finally_block? ) { D(fprintf(stderr, "%*c+ try_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'try' ':' block except_block+ else_block? finally_block?")); @@ -4536,7 +4536,7 @@ except_block_rule(Parser *p) && (e = expression_rule(p)) // expression && - (t = _tmp_49_rule(p), 1) // ['as' NAME] + (t = _tmp_49_rule(p), !p->error_indicator) // ['as' NAME] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -4688,7 +4688,7 @@ return_stmt_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 500)) // token='return' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ return_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'return' star_expressions?")); @@ -4753,7 +4753,7 @@ raise_stmt_rule(Parser *p) && (a = expression_rule(p)) // expression && - (b = _tmp_50_rule(p), 1) // ['from' expression] + (b = _tmp_50_rule(p), !p->error_indicator) // ['from' expression] ) { D(fprintf(stderr, "%*c+ raise_stmt[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'raise' expression ['from' expression]")); @@ -4924,15 +4924,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_51_rule(p), 1) // ['->' expression] + (a = _tmp_51_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -4984,15 +4984,15 @@ function_def_raw_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (params = params_rule(p), 1) // params? + (params = params_rule(p), !p->error_indicator) // params? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && - (a = _tmp_52_rule(p), 1) // ['->' expression] + (a = _tmp_52_rule(p), !p->error_indicator) // ['->' expression] && (_literal_2 = _PyPegen_expect_token(p, 11)) // token=':' && - (tc = func_type_comment_rule(p), 1) // func_type_comment? + (tc = func_type_comment_rule(p), !p->error_indicator) // func_type_comment? && (b = block_rule(p)) // block ) @@ -5200,7 +5200,7 @@ parameters_rule(Parser *p) && (c = _loop0_55_rule(p)) // param_with_default* && - (d = star_etc_rule(p), 1) // star_etc? + (d = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_no_default param_no_default* param_with_default* star_etc?")); @@ -5230,7 +5230,7 @@ parameters_rule(Parser *p) && (b = _loop0_56_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "slash_with_default param_with_default* star_etc?")); @@ -5260,7 +5260,7 @@ parameters_rule(Parser *p) && (b = _loop0_58_rule(p)) // param_with_default* && - (c = star_etc_rule(p), 1) // star_etc? + (c = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_no_default+ param_with_default* star_etc?")); @@ -5287,7 +5287,7 @@ parameters_rule(Parser *p) if ( (a = _loop1_59_rule(p)) // param_with_default+ && - (b = star_etc_rule(p), 1) // star_etc? + (b = star_etc_rule(p), !p->error_indicator) // star_etc? ) { D(fprintf(stderr, "%*c+ parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param_with_default+ star_etc?")); @@ -5525,7 +5525,7 @@ star_etc_rule(Parser *p) && (b = _loop0_66_rule(p)) // param_maybe_default* && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' param_no_default param_maybe_default* kwds?")); @@ -5558,7 +5558,7 @@ star_etc_rule(Parser *p) && (b = _loop1_67_rule(p)) // param_maybe_default+ && - (c = kwds_rule(p), 1) // kwds? + (c = kwds_rule(p), !p->error_indicator) // kwds? ) { D(fprintf(stderr, "%*c+ star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' param_maybe_default+ kwds?")); @@ -5692,7 +5692,7 @@ param_no_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_no_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param ',' TYPE_COMMENT?")); @@ -5719,7 +5719,7 @@ param_no_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -5771,7 +5771,7 @@ param_with_default_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_with_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default ',' TYPE_COMMENT?")); @@ -5801,7 +5801,7 @@ param_with_default_rule(Parser *p) && (c = default_rule(p)) // default && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -5851,11 +5851,11 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? ) { D(fprintf(stderr, "%*c+ param_maybe_default[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "param default? ',' TYPE_COMMENT?")); @@ -5883,9 +5883,9 @@ param_maybe_default_rule(Parser *p) if ( (a = param_rule(p)) // param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && - (tc = _PyPegen_expect_token(p, TYPE_COMMENT), 1) // TYPE_COMMENT? + (tc = _PyPegen_expect_token(p, TYPE_COMMENT), !p->error_indicator) // TYPE_COMMENT? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -5940,7 +5940,7 @@ param_rule(Parser *p) if ( (a = _PyPegen_name_token(p)) // NAME && - (b = annotation_rule(p), 1) // annotation? + (b = annotation_rule(p), !p->error_indicator) // annotation? ) { D(fprintf(stderr, "%*c+ param[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "NAME annotation?")); @@ -6199,7 +6199,7 @@ class_def_raw_rule(Parser *p) && (a = _PyPegen_name_token(p)) // NAME && - (b = _tmp_69_rule(p), 1) // ['(' arguments? ')'] + (b = _tmp_69_rule(p), !p->error_indicator) // ['(' arguments? ')'] && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -6365,7 +6365,7 @@ star_expressions_rule(Parser *p) && (b = _loop1_70_rule(p)) // ((',' star_expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_expression ((',' star_expression))+ ','?")); @@ -6560,7 +6560,7 @@ star_named_expressions_rule(Parser *p) if ( (a = _gather_71_rule(p)) // ','.star_named_expression+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_named_expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_named_expression+ ','?")); @@ -6865,7 +6865,7 @@ expressions_rule(Parser *p) && (b = _loop1_73_rule(p)) // ((',' expression))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ expressions[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression ((',' expression))+ ','?")); @@ -7098,7 +7098,7 @@ lambdef_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 525)) // token='lambda' && - (a = lambda_params_rule(p), 1) // lambda_params? + (a = lambda_params_rule(p), !p->error_indicator) // lambda_params? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && @@ -7221,7 +7221,7 @@ lambda_parameters_rule(Parser *p) && (c = _loop0_75_rule(p)) // lambda_param_with_default* && - (d = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (d = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_no_default lambda_param_no_default* lambda_param_with_default* lambda_star_etc?")); @@ -7251,7 +7251,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_76_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_slash_with_default lambda_param_with_default* lambda_star_etc?")); @@ -7281,7 +7281,7 @@ lambda_parameters_rule(Parser *p) && (b = _loop0_78_rule(p)) // lambda_param_with_default* && - (c = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (c = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_no_default+ lambda_param_with_default* lambda_star_etc?")); @@ -7308,7 +7308,7 @@ lambda_parameters_rule(Parser *p) if ( (a = _loop1_79_rule(p)) // lambda_param_with_default+ && - (b = lambda_star_etc_rule(p), 1) // lambda_star_etc? + (b = lambda_star_etc_rule(p), !p->error_indicator) // lambda_star_etc? ) { D(fprintf(stderr, "%*c+ lambda_parameters[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "lambda_param_with_default+ lambda_star_etc?")); @@ -7548,7 +7548,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop0_86_rule(p)) // lambda_param_maybe_default* && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' lambda_param_no_default lambda_param_maybe_default* lambda_kwds?")); @@ -7581,7 +7581,7 @@ lambda_star_etc_rule(Parser *p) && (b = _loop1_87_rule(p)) // lambda_param_maybe_default+ && - (c = lambda_kwds_rule(p), 1) // lambda_kwds? + (c = lambda_kwds_rule(p), !p->error_indicator) // lambda_kwds? ) { D(fprintf(stderr, "%*c+ lambda_star_etc[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'*' ',' lambda_param_maybe_default+ lambda_kwds?")); @@ -7859,7 +7859,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && (_literal = _PyPegen_expect_token(p, 12)) // token=',' ) @@ -7888,7 +7888,7 @@ lambda_param_maybe_default_rule(Parser *p) if ( (a = lambda_param_rule(p)) // lambda_param && - (c = default_rule(p), 1) // default? + (c = default_rule(p), !p->error_indicator) // default? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 11) // token=':' ) @@ -10404,7 +10404,7 @@ primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -10556,7 +10556,7 @@ slices_rule(Parser *p) if ( (a = _gather_92_rule(p)) // ','.slice+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ slices[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.slice+ ','?")); @@ -10618,13 +10618,13 @@ slice_rule(Parser *p) void *b; void *c; if ( - (a = expression_rule(p), 1) // expression? + (a = expression_rule(p), !p->error_indicator) // expression? && (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (b = expression_rule(p), 1) // expression? + (b = expression_rule(p), !p->error_indicator) // expression? && - (c = _tmp_94_rule(p), 1) // [':' expression?] + (c = _tmp_94_rule(p), !p->error_indicator) // [':' expression?] ) { D(fprintf(stderr, "%*c+ slice[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression? ':' expression? [':' expression?]")); @@ -11072,7 +11072,7 @@ list_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_named_expressions_rule(p), 1) // star_named_expressions? + (a = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -11231,7 +11231,7 @@ tuple_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = _tmp_99_rule(p), 1) // [star_named_expression ',' star_named_expressions?] + (a = _tmp_99_rule(p), !p->error_indicator) // [star_named_expression ',' star_named_expressions?] && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -11615,7 +11615,7 @@ dict_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 25)) // token='{' && - (a = double_starred_kvpairs_rule(p), 1) // double_starred_kvpairs? + (a = double_starred_kvpairs_rule(p), !p->error_indicator) // double_starred_kvpairs? && (_literal_1 = _PyPegen_expect_token(p, 26)) // token='}' ) @@ -11758,7 +11758,7 @@ double_starred_kvpairs_rule(Parser *p) if ( (a = _gather_101_rule(p)) // ','.double_starred_kvpair+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ double_starred_kvpairs[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.double_starred_kvpair+ ','?")); @@ -12124,7 +12124,7 @@ yield_expr_rule(Parser *p) if ( (_keyword = _PyPegen_expect_token(p, 504)) // token='yield' && - (a = star_expressions_rule(p), 1) // star_expressions? + (a = star_expressions_rule(p), !p->error_indicator) // star_expressions? ) { D(fprintf(stderr, "%*c+ yield_expr[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "'yield' star_expressions?")); @@ -12182,7 +12182,7 @@ arguments_rule(Parser *p) if ( (a = args_rule(p)) // args && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? && _PyPegen_lookahead_with_int(1, _PyPegen_expect_token, p, 8) // token=')' ) @@ -12257,7 +12257,7 @@ args_rule(Parser *p) if ( (a = _gather_106_rule(p)) // ','.(starred_expression | named_expression !'=')+ && - (b = _tmp_108_rule(p), 1) // [',' kwargs] + (b = _tmp_108_rule(p), !p->error_indicator) // [',' kwargs] ) { D(fprintf(stderr, "%*c+ args[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.(starred_expression | named_expression !'=')+ [',' kwargs]")); @@ -12760,7 +12760,7 @@ star_targets_rule(Parser *p) && (b = _loop0_117_rule(p)) // ((',' star_target))* && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))* ','?")); @@ -12814,7 +12814,7 @@ star_targets_list_seq_rule(Parser *p) if ( (a = _gather_118_rule(p)) // ','.star_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_list_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.star_target+ ','?")); @@ -12862,7 +12862,7 @@ star_targets_tuple_seq_rule(Parser *p) && (b = _loop1_120_rule(p)) // ((',' star_target))+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ star_targets_tuple_seq[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_target ((',' star_target))+ ','?")); @@ -13225,7 +13225,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = star_targets_tuple_seq_rule(p), 1) // star_targets_tuple_seq? + (a = star_targets_tuple_seq_rule(p), !p->error_indicator) // star_targets_tuple_seq? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -13264,7 +13264,7 @@ star_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = star_targets_list_seq_rule(p), 1) // star_targets_list_seq? + (a = star_targets_list_seq_rule(p), !p->error_indicator) // star_targets_list_seq? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -13523,7 +13523,7 @@ del_targets_rule(Parser *p) if ( (a = _gather_122_rule(p)) // ','.del_target+ && - (_opt_var = _PyPegen_expect_token(p, 12), 1) // ','? + (_opt_var = _PyPegen_expect_token(p, 12), !p->error_indicator) // ','? ) { D(fprintf(stderr, "%*c+ del_targets[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "','.del_target+ ','?")); @@ -13769,7 +13769,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -13808,7 +13808,7 @@ del_t_atom_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 9)) // token='[' && - (a = del_targets_rule(p), 1) // del_targets? + (a = del_targets_rule(p), !p->error_indicator) // del_targets? && (_literal_1 = _PyPegen_expect_token(p, 10)) // token=']' ) @@ -14036,7 +14036,7 @@ t_primary_raw(Parser *p) && (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (b = arguments_rule(p), 1) // arguments? + (b = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' && @@ -14235,7 +14235,7 @@ invalid_arguments_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (_opt_var = _tmp_124_rule(p), 1) // [args | expression for_if_clauses] + (_opt_var = _tmp_124_rule(p), !p->error_indicator) // [args | expression for_if_clauses] ) { D(fprintf(stderr, "%*c+ invalid_arguments[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "expression for_if_clauses ',' [args | expression for_if_clauses]")); @@ -15302,7 +15302,7 @@ invalid_for_target_rule(Parser *p) UNUSED(_opt_var); // Silence compiler warnings expr_ty a; if ( - (_opt_var = _PyPegen_expect_token(p, ASYNC), 1) // ASYNC? + (_opt_var = _PyPegen_expect_token(p, ASYNC), !p->error_indicator) // ASYNC? && (_keyword = _PyPegen_expect_token(p, 517)) // token='for' && @@ -19473,7 +19473,7 @@ _tmp_69_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 7)) // token='(' && - (z = arguments_rule(p), 1) // arguments? + (z = arguments_rule(p), !p->error_indicator) // arguments? && (_literal_1 = _PyPegen_expect_token(p, 8)) // token=')' ) @@ -21102,7 +21102,7 @@ _tmp_94_rule(Parser *p) if ( (_literal = _PyPegen_expect_token(p, 11)) // token=':' && - (d = expression_rule(p), 1) // expression? + (d = expression_rule(p), !p->error_indicator) // expression? ) { D(fprintf(stderr, "%*c+ _tmp_94[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "':' expression?")); @@ -21442,7 +21442,7 @@ _tmp_99_rule(Parser *p) && (_literal = _PyPegen_expect_token(p, 12)) // token=',' && - (z = star_named_expressions_rule(p), 1) // star_named_expressions? + (z = star_named_expressions_rule(p), !p->error_indicator) // star_named_expressions? ) { D(fprintf(stderr, "%*c+ _tmp_99[%d-%d]: %s succeeded!\n", p->level, ' ', _mark, p->mark, "star_named_expression ',' star_named_expressions?")); diff --git a/Parser/pegen/pegen.c b/Parser/pegen/pegen.c index bf53214b4d063..98de05c5cdb69 100644 --- a/Parser/pegen/pegen.c +++ b/Parser/pegen/pegen.c @@ -347,10 +347,18 @@ tokenizer_error(Parser *p) errtype = PyExc_IndentationError; msg = "too many levels of indentation"; break; - case E_LINECONT: - col_offset = strlen(strtok(p->tok->buf, "\n")) - 1; + case E_LINECONT: { + char* loc = strrchr(p->tok->buf, '\n'); + const char* last_char = p->tok->cur - 1; + if (loc != NULL && loc != last_char) { + col_offset = p->tok->cur - loc - 1; + p->tok->buf = loc; + } else { + col_offset = last_char - p->tok->buf - 1; + } msg = "unexpected character after line continuation character"; break; + } default: msg = "unknown parsing error"; } diff --git a/Tools/peg_generator/pegen/c_generator.py b/Tools/peg_generator/pegen/c_generator.py index 692ee69031ba5..b5836f62263ef 100644 --- a/Tools/peg_generator/pegen/c_generator.py +++ b/Tools/peg_generator/pegen/c_generator.py @@ -85,7 +85,7 @@ def __str__(self) -> str: if self.arguments: parts.append(f"({', '.join(map(str, self.arguments))})") if self.force_true: - parts.append(", 1") + parts.append(", !p->error_indicator") if self.assigned_variable: parts = ["(", self.assigned_variable, " = ", *parts, ")"] if self.comment: From webhook-mailer at python.org Wed Oct 20 12:52:32 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 16:52:32 -0000 Subject: [Python-checkins] bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29095) Message-ID: https://github.com/python/cpython/commit/65de808811f93793599209f74bb1bab3ad399b17 commit: 65de808811f93793599209f74bb1bab3ad399b17 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T18:52:22+02:00 summary: bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29095) (cherry picked from commit 8d6740f489fca67a44de165d29d9e0ad86285779) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_doctest.py M Lib/test/test_doctest2.py diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 828a0ff56763a..7f8ccd3896ead 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -3110,20 +3110,11 @@ def test_no_trailing_whitespace_stripping(): patches that contain trailing whitespace. More info on Issue 24746. """ -###################################################################### -## Main -###################################################################### - -def test_main(): - # Check the doctest cases in doctest itself: - ret = support.run_doctest(doctest, verbosity=True) - # Check the doctest cases defined here: - from test import test_doctest - support.run_doctest(test_doctest, verbosity=True) - - # Run unittests - support.run_unittest(__name__) +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(doctest)) + tests.addTest(doctest.DocTestSuite()) + return tests def test_coverage(coverdir): @@ -3136,8 +3127,9 @@ def test_coverage(coverdir): r.write_results(show_missing=True, summary=True, coverdir=coverdir) + if __name__ == '__main__': if '-c' in sys.argv: test_coverage('/tmp/doctest.cover') else: - test_main() + unittest.main() diff --git a/Lib/test/test_doctest2.py b/Lib/test/test_doctest2.py index 347a143641071..ab8a0696736e2 100644 --- a/Lib/test/test_doctest2.py +++ b/Lib/test/test_doctest2.py @@ -13,7 +13,6 @@ import sys import unittest -from test import support if sys.flags.optimize >= 2: raise unittest.SkipTest("Cannot test docstrings with -O2") @@ -107,17 +106,21 @@ def clsm(cls, val): """ return val -def test_main(): - from test import test_doctest2 - EXPECTED = 19 - f, t = support.run_doctest(test_doctest2) - if t != EXPECTED: - raise support.TestFailed("expected %d tests to run, not %d" % - (EXPECTED, t)) + +class Test(unittest.TestCase): + def test_testmod(self): + import doctest, sys + EXPECTED = 19 + f, t = doctest.testmod(sys.modules[__name__]) + if f: + self.fail("%d of %d doctests failed" % (f, t)) + if t != EXPECTED: + self.fail("expected %d tests to run, not %d" % (EXPECTED, t)) + # Pollute the namespace with a bunch of imported functions and classes, # to make sure they don't get tested. from doctest import * if __name__ == '__main__': - test_main() + unittest.main() From webhook-mailer at python.org Wed Oct 20 12:52:46 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 16:52:46 -0000 Subject: [Python-checkins] bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29096) Message-ID: https://github.com/python/cpython/commit/919268316582c6ac47960c2e5dd2ee1682371494 commit: 919268316582c6ac47960c2e5dd2ee1682371494 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T18:52:41+02:00 summary: bpo-45229: Make doctest tests discoverable (GH-28986) (GH-29096) (cherry picked from commit 8d6740f489fca67a44de165d29d9e0ad86285779) Co-authored-by: Serhiy Storchaka files: M Lib/test/test_doctest.py M Lib/test/test_doctest2.py diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 3efe5dafc20ad..af5513c631777 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -3091,20 +3091,11 @@ def test_no_trailing_whitespace_stripping(): patches that contain trailing whitespace. More info on Issue 24746. """ -###################################################################### -## Main -###################################################################### - -def test_main(): - # Check the doctest cases in doctest itself: - ret = support.run_doctest(doctest, verbosity=True) - # Check the doctest cases defined here: - from test import test_doctest - support.run_doctest(test_doctest, verbosity=True) - - # Run unittests - support.run_unittest(__name__) +def load_tests(loader, tests, pattern): + tests.addTest(doctest.DocTestSuite(doctest)) + tests.addTest(doctest.DocTestSuite()) + return tests def test_coverage(coverdir): @@ -3117,8 +3108,9 @@ def test_coverage(coverdir): r.write_results(show_missing=True, summary=True, coverdir=coverdir) + if __name__ == '__main__': if '-c' in sys.argv: test_coverage('/tmp/doctest.cover') else: - test_main() + unittest.main() diff --git a/Lib/test/test_doctest2.py b/Lib/test/test_doctest2.py index 347a143641071..ab8a0696736e2 100644 --- a/Lib/test/test_doctest2.py +++ b/Lib/test/test_doctest2.py @@ -13,7 +13,6 @@ import sys import unittest -from test import support if sys.flags.optimize >= 2: raise unittest.SkipTest("Cannot test docstrings with -O2") @@ -107,17 +106,21 @@ def clsm(cls, val): """ return val -def test_main(): - from test import test_doctest2 - EXPECTED = 19 - f, t = support.run_doctest(test_doctest2) - if t != EXPECTED: - raise support.TestFailed("expected %d tests to run, not %d" % - (EXPECTED, t)) + +class Test(unittest.TestCase): + def test_testmod(self): + import doctest, sys + EXPECTED = 19 + f, t = doctest.testmod(sys.modules[__name__]) + if f: + self.fail("%d of %d doctests failed" % (f, t)) + if t != EXPECTED: + self.fail("expected %d tests to run, not %d" % (EXPECTED, t)) + # Pollute the namespace with a bunch of imported functions and classes, # to make sure they don't get tested. from doctest import * if __name__ == '__main__': - test_main() + unittest.main() From webhook-mailer at python.org Wed Oct 20 12:54:37 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 16:54:37 -0000 Subject: [Python-checkins] bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) Message-ID: https://github.com/python/cpython/commit/dff0b713436e286bb1afdd7c6f3093c8e8db16dd commit: dff0b713436e286bb1afdd7c6f3093c8e8db16dd branch: main author: ?ukasz Langa committer: ambv date: 2021-10-20T18:54:31+02:00 summary: bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) Co-authored-by: Pablo Galindo Salgado files: A Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst M Doc/library/exceptions.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index b665c6079ab0a..8fa82a98a199d 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -34,6 +34,10 @@ class or one of its subclasses, and not from :exc:`BaseException`. More information on defining exceptions is available in the Python Tutorial under :ref:`tut-userexceptions`. + +Exception context +----------------- + When raising (or re-raising) an exception in an :keyword:`except` or :keyword:`finally` clause :attr:`__context__` is automatically set to the last exception caught; if the @@ -67,6 +71,25 @@ exceptions so that the final line of the traceback always shows the last exception that was raised. +Inheriting from built-in exceptions +----------------------------------- + +User code can create subclasses that inherit from an exception type. +It's recommended to only subclass one exception type at a time to avoid +any possible conflicts between how the bases handle the ``args`` +attribute, as well as due to possible memory layout incompatibilities. + +.. impl-detail:: + + Most built-in exceptions are implemented in C for efficiency, see: + :source:`Objects/exceptions.c`. Some have custom memory layouts + which makes it impossible to create a subclass that inherits from + multiple exception types. The memory layout of a type is an implementation + detail and might change between Python versions, leading to new + conflicts in the future. Therefore, it's recommended to avoid + subclassing multiple exception types altogether. + + Base classes ------------ diff --git a/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst new file mode 100644 index 0000000000000..1721aa2c2dfc4 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst @@ -0,0 +1,4 @@ +Mention in the documentation of :ref:`Built-in Exceptions +` that inheriting from multiple exception types in a +single subclass is not recommended due to possible memory layout +incompatibility. From webhook-mailer at python.org Wed Oct 20 13:09:13 2021 From: webhook-mailer at python.org (corona10) Date: Wed, 20 Oct 2021 17:09:13 -0000 Subject: [Python-checkins] bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) Message-ID: https://github.com/python/cpython/commit/d2cd5eef0c3fc0431bfe3fc24b4c020ebfcf8aad commit: d2cd5eef0c3fc0431bfe3fc24b4c020ebfcf8aad branch: main author: Jeong YunWon <69878+youknowone at users.noreply.github.com> committer: corona10 date: 2021-10-21T02:09:03+09:00 summary: bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) files: A Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst M Modules/getbuildinfo.c diff --git a/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst new file mode 100644 index 0000000000000..575e2fb9ae93b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst @@ -0,0 +1,2 @@ +Update :data:`sys.version` to use ``main`` as fallback information. +Patch by Jeong YunWon. diff --git a/Modules/getbuildinfo.c b/Modules/getbuildinfo.c index 5f941a26e1d54..7cb7397a22c8a 100644 --- a/Modules/getbuildinfo.c +++ b/Modules/getbuildinfo.c @@ -40,8 +40,9 @@ Py_GetBuildInfo(void) const char *revision = _Py_gitversion(); const char *sep = *revision ? ":" : ""; const char *gitid = _Py_gitidentifier(); - if (!(*gitid)) - gitid = "default"; + if (!(*gitid)) { + gitid = "main"; + } PyOS_snprintf(buildinfo, sizeof(buildinfo), "%s%s%s, %.20s, %.9s", gitid, sep, revision, DATE, TIME); From webhook-mailer at python.org Wed Oct 20 13:15:08 2021 From: webhook-mailer at python.org (iritkatriel) Date: Wed, 20 Oct 2021 17:15:08 -0000 Subject: [Python-checkins] bpo-41374: Remove obsolete exclusion of netinet/tcp.h on Cygwin (GH-21649) Message-ID: https://github.com/python/cpython/commit/d8e181925123ab1fd3dfcad3b29325b2b0ff704b commit: d8e181925123ab1fd3dfcad3b29325b2b0ff704b branch: main author: Zackery Spytz committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2021-10-20T18:14:59+01:00 summary: bpo-41374: Remove obsolete exclusion of netinet/tcp.h on Cygwin (GH-21649) files: A Misc/NEWS.d/next/Library/2020-07-27-19-21-05.bpo-41374.cd-kFL.rst M Modules/socketmodule.h diff --git a/Misc/NEWS.d/next/Library/2020-07-27-19-21-05.bpo-41374.cd-kFL.rst b/Misc/NEWS.d/next/Library/2020-07-27-19-21-05.bpo-41374.cd-kFL.rst new file mode 100644 index 0000000000000..a5b2e042a3fae --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-07-27-19-21-05.bpo-41374.cd-kFL.rst @@ -0,0 +1,2 @@ +Ensure that ``socket.TCP_*`` constants are exposed on Cygwin 3.1.6 and +greater. diff --git a/Modules/socketmodule.h b/Modules/socketmodule.h index e4f375d5e8100..aea599f0ee6c8 100644 --- a/Modules/socketmodule.h +++ b/Modules/socketmodule.h @@ -8,9 +8,7 @@ # include # endif # include -# if !defined(__CYGWIN__) -# include -# endif +# include #else /* MS_WINDOWS */ # include From webhook-mailer at python.org Wed Oct 20 13:41:50 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 17:41:50 -0000 Subject: [Python-checkins] bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) Message-ID: https://github.com/python/cpython/commit/696a89fef81f8229ebff9c32dfd36921c04f0890 commit: 696a89fef81f8229ebff9c32dfd36921c04f0890 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T10:41:35-07:00 summary: bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) (cherry picked from commit d2cd5eef0c3fc0431bfe3fc24b4c020ebfcf8aad) Co-authored-by: Jeong YunWon <69878+youknowone at users.noreply.github.com> files: A Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst M Modules/getbuildinfo.c diff --git a/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst new file mode 100644 index 0000000000000..575e2fb9ae93b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst @@ -0,0 +1,2 @@ +Update :data:`sys.version` to use ``main`` as fallback information. +Patch by Jeong YunWon. diff --git a/Modules/getbuildinfo.c b/Modules/getbuildinfo.c index 5f941a26e1d54..7cb7397a22c8a 100644 --- a/Modules/getbuildinfo.c +++ b/Modules/getbuildinfo.c @@ -40,8 +40,9 @@ Py_GetBuildInfo(void) const char *revision = _Py_gitversion(); const char *sep = *revision ? ":" : ""; const char *gitid = _Py_gitidentifier(); - if (!(*gitid)) - gitid = "default"; + if (!(*gitid)) { + gitid = "main"; + } PyOS_snprintf(buildinfo, sizeof(buildinfo), "%s%s%s, %.20s, %.9s", gitid, sep, revision, DATE, TIME); From webhook-mailer at python.org Wed Oct 20 13:45:14 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 20 Oct 2021 17:45:14 -0000 Subject: [Python-checkins] bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) Message-ID: https://github.com/python/cpython/commit/1249ce7c6c0ac7a99a72e6e7b8b10dd64158c386 commit: 1249ce7c6c0ac7a99a72e6e7b8b10dd64158c386 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T10:45:06-07:00 summary: bpo-45532: Replace 'default' with 'main' as default in sys.version (GH-29100) (cherry picked from commit d2cd5eef0c3fc0431bfe3fc24b4c020ebfcf8aad) Co-authored-by: Jeong YunWon <69878+youknowone at users.noreply.github.com> files: A Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst M Modules/getbuildinfo.c diff --git a/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst new file mode 100644 index 0000000000000..575e2fb9ae93b --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-16-07-39.bpo-45532.kyhvis.rst @@ -0,0 +1,2 @@ +Update :data:`sys.version` to use ``main`` as fallback information. +Patch by Jeong YunWon. diff --git a/Modules/getbuildinfo.c b/Modules/getbuildinfo.c index 5f941a26e1d54..7cb7397a22c8a 100644 --- a/Modules/getbuildinfo.c +++ b/Modules/getbuildinfo.c @@ -40,8 +40,9 @@ Py_GetBuildInfo(void) const char *revision = _Py_gitversion(); const char *sep = *revision ? ":" : ""; const char *gitid = _Py_gitidentifier(); - if (!(*gitid)) - gitid = "default"; + if (!(*gitid)) { + gitid = "main"; + } PyOS_snprintf(buildinfo, sizeof(buildinfo), "%s%s%s, %.20s, %.9s", gitid, sep, revision, DATE, TIME); From webhook-mailer at python.org Wed Oct 20 14:49:04 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 18:49:04 -0000 Subject: [Python-checkins] bpo-45320: Remove long-deprecated inspect methods (GH-28618) Message-ID: https://github.com/python/cpython/commit/d89fb9a5a610a257014d112bdceef73d7df14082 commit: d89fb9a5a610a257014d112bdceef73d7df14082 branch: main author: Hugo van Kemenade committer: ambv date: 2021-10-20T20:48:55+02:00 summary: bpo-45320: Remove long-deprecated inspect methods (GH-28618) files: A Misc/NEWS.d/next/Library/2021-09-15-10-21-10.bpo-45320.4qaf5x.rst M Doc/howto/clinic.rst M Doc/library/inspect.rst M Doc/whatsnew/3.11.rst M Lib/inspect.py M Lib/test/test_inspect.py diff --git a/Doc/howto/clinic.rst b/Doc/howto/clinic.rst index 3a3653a5ee3a3..a3c4330296bed 100644 --- a/Doc/howto/clinic.rst +++ b/Doc/howto/clinic.rst @@ -567,9 +567,6 @@ expression. Currently the following are explicitly supported: * Simple symbolic constants like ``sys.maxsize``, which must start with the name of the module -In case you're curious, this is implemented in ``from_builtin()`` -in ``Lib/inspect.py``. - (In the future, this may need to get even more elaborate, to allow full expressions like ``CONSTANT - 1``.) diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst index ed95c72c9a626..1074f977184bb 100644 --- a/Doc/library/inspect.rst +++ b/Doc/library/inspect.rst @@ -935,26 +935,6 @@ Classes and functions times. -.. function:: getargspec(func) - - Get the names and default values of a Python function's parameters. A - :term:`named tuple` ``ArgSpec(args, varargs, keywords, defaults)`` is - returned. *args* is a list of the parameter names. *varargs* and *keywords* - are the names of the ``*`` and ``**`` parameters or ``None``. *defaults* is a - tuple of default argument values or ``None`` if there are no default - arguments; if this tuple has *n* elements, they correspond to the last - *n* elements listed in *args*. - - .. deprecated:: 3.0 - Use :func:`getfullargspec` for an updated API that is usually a drop-in - replacement, but also correctly handles function annotations and - keyword-only parameters. - - Alternatively, use :func:`signature` and - :ref:`Signature Object `, which provide a - more structured introspection API for callables. - - .. function:: getfullargspec(func) Get the names and default values of a Python function's parameters. A @@ -1015,33 +995,6 @@ Classes and functions This function was inadvertently marked as deprecated in Python 3.5. -.. function:: formatargspec(args[, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations[, formatarg, formatvarargs, formatvarkw, formatvalue, formatreturns, formatannotations]]) - - Format a pretty argument spec from the values returned by - :func:`getfullargspec`. - - The first seven arguments are (``args``, ``varargs``, ``varkw``, - ``defaults``, ``kwonlyargs``, ``kwonlydefaults``, ``annotations``). - - The other six arguments are functions that are called to turn argument names, - ``*`` argument name, ``**`` argument name, default values, return annotation - and individual annotations into strings, respectively. - - For example: - - >>> from inspect import formatargspec, getfullargspec - >>> def f(a: int, b: float): - ... pass - ... - >>> formatargspec(*getfullargspec(f)) - '(a: int, b: float)' - - .. deprecated:: 3.5 - Use :func:`signature` and - :ref:`Signature Object `, which provide a - better introspecting API for callables. - - .. function:: formatargvalues(args[, varargs, varkw, locals, formatarg, formatvarargs, formatvarkw, formatvalue]) Format a pretty argument spec from the four values returned by diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 13c1e72306653..d5e8bc967a816 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -363,7 +363,7 @@ Removed ``SO_REUSEADDR`` in UDP. (Contributed by Hugo van Kemenade in :issue:`45129`.) -* Remove :meth:`__getitem__` methods of +* Removed :meth:`__getitem__` methods of :class:`xml.dom.pulldom.DOMEventStream`, :class:`wsgiref.util.FileWrapper` and :class:`fileinput.FileInput`, deprecated since Python 3.9. (Contributed by Hugo van Kemenade in :issue:`45132`.) @@ -402,7 +402,7 @@ Removed the ``l*gettext()`` functions. (Contributed by Dong-hee Na and Serhiy Storchaka in :issue:`44235`.) -* Remove from the :mod:`configparser` module: +* Removed from the :mod:`configparser` module: the :class:`SafeConfigParser` class, the :attr:`filename` property of the :class:`ParsingError` class, the :meth:`readfp` method of the :class:`ConfigParser` class, @@ -419,9 +419,25 @@ Removed generator-based coroutine objects in the debug mode. (Contributed by Illia Volochii in :issue:`43216`.) -* Remove the deprecated ``split()`` method of :class:`_tkinter.TkappType`. +* Removed the deprecated ``split()`` method of :class:`_tkinter.TkappType`. (Contributed by Erlend E. Aasland in :issue:`38371`.) +* Removed from the :mod:`inspect` module: + + * the ``getargspec`` function, deprecated since Python 3.0; + use :func:`inspect.signature` or :func:`inspect.getfullargspec` instead. + + * the ``formatargspec`` function, deprecated since Python 3.5; + use the :func:`inspect.signature` function and :class:`Signature` object + directly. + + * the undocumented ``Signature.from_callable`` and ``Signature.from_function`` + functions, deprecated since Python 3.5; use the + :meth:`Signature.from_callable() ` method + instead. + + (Contributed by Hugo van Kemenade in :issue:`45320`.) + Porting to Python 3.11 ====================== diff --git a/Lib/inspect.py b/Lib/inspect.py index 1b2fc918c8131..a956acf310f85 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -47,7 +47,6 @@ import tokenize import token import types -import warnings import functools import builtins from operator import attrgetter @@ -1214,37 +1213,6 @@ def getargs(co): varkw = co.co_varnames[nargs] return Arguments(args + kwonlyargs, varargs, varkw) -ArgSpec = namedtuple('ArgSpec', 'args varargs keywords defaults') - -def getargspec(func): - """Get the names and default values of a function's parameters. - - A tuple of four things is returned: (args, varargs, keywords, defaults). - 'args' is a list of the argument names, including keyword-only argument names. - 'varargs' and 'keywords' are the names of the * and ** parameters or None. - 'defaults' is an n-tuple of the default values of the last n parameters. - - This function is deprecated, as it does not support annotations or - keyword-only parameters and will raise ValueError if either is present - on the supplied callable. - - For a more structured introspection API, use inspect.signature() instead. - - Alternatively, use getfullargspec() for an API with a similar namedtuple - based interface, but full support for annotations and keyword-only - parameters. - - Deprecated since Python 3.5, use `inspect.getfullargspec()`. - """ - warnings.warn("inspect.getargspec() is deprecated since Python 3.0, " - "use inspect.signature() or inspect.getfullargspec()", - DeprecationWarning, stacklevel=2) - args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, ann = \ - getfullargspec(func) - if kwonlyargs or ann: - raise ValueError("Function has keyword-only parameters or annotations" - ", use inspect.signature() API which can support them") - return ArgSpec(args, varargs, varkw, defaults) FullArgSpec = namedtuple('FullArgSpec', 'args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, annotations') @@ -1369,63 +1337,6 @@ def _formatannotation(annotation): return formatannotation(annotation, module) return _formatannotation -def formatargspec(args, varargs=None, varkw=None, defaults=None, - kwonlyargs=(), kwonlydefaults={}, annotations={}, - formatarg=str, - formatvarargs=lambda name: '*' + name, - formatvarkw=lambda name: '**' + name, - formatvalue=lambda value: '=' + repr(value), - formatreturns=lambda text: ' -> ' + text, - formatannotation=formatannotation): - """Format an argument spec from the values returned by getfullargspec. - - The first seven arguments are (args, varargs, varkw, defaults, - kwonlyargs, kwonlydefaults, annotations). The other five arguments - are the corresponding optional formatting functions that are called to - turn names and values into strings. The last argument is an optional - function to format the sequence of arguments. - - Deprecated since Python 3.5: use the `signature` function and `Signature` - objects. - """ - - from warnings import warn - - warn("`formatargspec` is deprecated since Python 3.5. Use `signature` and " - "the `Signature` object directly", - DeprecationWarning, - stacklevel=2) - - def formatargandannotation(arg): - result = formatarg(arg) - if arg in annotations: - result += ': ' + formatannotation(annotations[arg]) - return result - specs = [] - if defaults: - firstdefault = len(args) - len(defaults) - for i, arg in enumerate(args): - spec = formatargandannotation(arg) - if defaults and i >= firstdefault: - spec = spec + formatvalue(defaults[i - firstdefault]) - specs.append(spec) - if varargs is not None: - specs.append(formatvarargs(formatargandannotation(varargs))) - else: - if kwonlyargs: - specs.append('*') - if kwonlyargs: - for kwonlyarg in kwonlyargs: - spec = formatargandannotation(kwonlyarg) - if kwonlydefaults and kwonlyarg in kwonlydefaults: - spec += formatvalue(kwonlydefaults[kwonlyarg]) - specs.append(spec) - if varkw is not None: - specs.append(formatvarkw(formatargandannotation(varkw))) - result = '(' + ', '.join(specs) + ')' - if 'return' in annotations: - result += formatreturns(formatannotation(annotations['return'])) - return result def formatargvalues(args, varargs, varkw, locals, formatarg=str, @@ -2932,30 +2843,6 @@ def __init__(self, parameters=None, *, return_annotation=_empty, self._parameters = types.MappingProxyType(params) self._return_annotation = return_annotation - @classmethod - def from_function(cls, func): - """Constructs Signature for the given python function. - - Deprecated since Python 3.5, use `Signature.from_callable()`. - """ - - warnings.warn("inspect.Signature.from_function() is deprecated since " - "Python 3.5, use Signature.from_callable()", - DeprecationWarning, stacklevel=2) - return _signature_from_function(cls, func) - - @classmethod - def from_builtin(cls, func): - """Constructs Signature for the given builtin function. - - Deprecated since Python 3.5, use `Signature.from_callable()`. - """ - - warnings.warn("inspect.Signature.from_builtin() is deprecated since " - "Python 3.5, use Signature.from_callable()", - DeprecationWarning, stacklevel=2) - return _signature_from_builtin(cls, func) - @classmethod def from_callable(cls, obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False): diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 0e9191e5073cc..516efac45e523 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -43,7 +43,7 @@ # ismodule, isclass, ismethod, isfunction, istraceback, isframe, iscode, # isbuiltin, isroutine, isgenerator, isgeneratorfunction, getmembers, # getdoc, getfile, getmodule, getsourcefile, getcomments, getsource, -# getclasstree, getargvalues, formatargspec, formatargvalues, +# getclasstree, getargvalues, formatargvalues, # currentframe, stack, trace, isdatadescriptor # NOTE: There are some additional tests relating to interaction with @@ -844,24 +844,11 @@ class D(B, C): pass got = inspect.getmro(D) self.assertEqual(expected, got) - def assertArgSpecEquals(self, routine, args_e, varargs_e=None, - varkw_e=None, defaults_e=None, formatted=None): - with self.assertWarns(DeprecationWarning): - args, varargs, varkw, defaults = inspect.getargspec(routine) - self.assertEqual(args, args_e) - self.assertEqual(varargs, varargs_e) - self.assertEqual(varkw, varkw_e) - self.assertEqual(defaults, defaults_e) - if formatted is not None: - with self.assertWarns(DeprecationWarning): - self.assertEqual(inspect.formatargspec(args, varargs, varkw, defaults), - formatted) - def assertFullArgSpecEquals(self, routine, args_e, varargs_e=None, varkw_e=None, defaults_e=None, posonlyargs_e=[], kwonlyargs_e=[], kwonlydefaults_e=None, - ann_e={}, formatted=None): + ann_e={}): args, varargs, varkw, defaults, kwonlyargs, kwonlydefaults, ann = \ inspect.getfullargspec(routine) self.assertEqual(args, args_e) @@ -871,58 +858,30 @@ def assertFullArgSpecEquals(self, routine, args_e, varargs_e=None, self.assertEqual(kwonlyargs, kwonlyargs_e) self.assertEqual(kwonlydefaults, kwonlydefaults_e) self.assertEqual(ann, ann_e) - if formatted is not None: - with self.assertWarns(DeprecationWarning): - self.assertEqual(inspect.formatargspec(args, varargs, varkw, defaults, - kwonlyargs, kwonlydefaults, ann), - formatted) - - def test_getargspec(self): - self.assertArgSpecEquals(mod.eggs, ['x', 'y'], formatted='(x, y)') - - self.assertArgSpecEquals(mod.spam, - ['a', 'b', 'c', 'd', 'e', 'f'], - 'g', 'h', (3, 4, 5), - '(a, b, c, d=3, e=4, f=5, *g, **h)') - - self.assertRaises(ValueError, self.assertArgSpecEquals, - mod2.keyworded, []) - - self.assertRaises(ValueError, self.assertArgSpecEquals, - mod2.annotated, []) - self.assertRaises(ValueError, self.assertArgSpecEquals, - mod2.keyword_only_arg, []) - def test_getfullargspec(self): self.assertFullArgSpecEquals(mod2.keyworded, [], varargs_e='arg1', kwonlyargs_e=['arg2'], - kwonlydefaults_e={'arg2':1}, - formatted='(*arg1, arg2=1)') + kwonlydefaults_e={'arg2':1}) self.assertFullArgSpecEquals(mod2.annotated, ['arg1'], - ann_e={'arg1' : list}, - formatted='(arg1: list)') + ann_e={'arg1' : list}) self.assertFullArgSpecEquals(mod2.keyword_only_arg, [], - kwonlyargs_e=['arg'], - formatted='(*, arg)') + kwonlyargs_e=['arg']) self.assertFullArgSpecEquals(mod2.all_markers, ['a', 'b', 'c', 'd'], - kwonlyargs_e=['e', 'f'], - formatted='(a, b, c, d, *, e, f)') + kwonlyargs_e=['e', 'f']) self.assertFullArgSpecEquals(mod2.all_markers_with_args_and_kwargs, ['a', 'b', 'c', 'd'], varargs_e='args', varkw_e='kwargs', - kwonlyargs_e=['e', 'f'], - formatted='(a, b, c, d, *args, e, f, **kwargs)') + kwonlyargs_e=['e', 'f']) self.assertFullArgSpecEquals(mod2.all_markers_with_defaults, ['a', 'b', 'c', 'd'], defaults_e=(1,2,3), kwonlyargs_e=['e', 'f'], - kwonlydefaults_e={'e': 4, 'f': 5}, - formatted='(a, b=1, c=2, d=3, *, e=4, f=5)') + kwonlydefaults_e={'e': 4, 'f': 5}) def test_argspec_api_ignores_wrapped(self): # Issue 20684: low level introspection API must ignore __wrapped__ @@ -930,39 +889,9 @@ def test_argspec_api_ignores_wrapped(self): def ham(x, y): pass # Basic check - self.assertArgSpecEquals(ham, ['x', 'y'], formatted='(x, y)') - self.assertFullArgSpecEquals(ham, ['x', 'y'], formatted='(x, y)') + self.assertFullArgSpecEquals(ham, ['x', 'y']) self.assertFullArgSpecEquals(functools.partial(ham), - ['x', 'y'], formatted='(x, y)') - # Other variants - def check_method(f): - self.assertArgSpecEquals(f, ['self', 'x', 'y'], - formatted='(self, x, y)') - class C: - @functools.wraps(mod.spam) - def ham(self, x, y): - pass - pham = functools.partialmethod(ham) - @functools.wraps(mod.spam) - def __call__(self, x, y): - pass - check_method(C()) - check_method(C.ham) - check_method(C().ham) - check_method(C.pham) - check_method(C().pham) - - class C_new: - @functools.wraps(mod.spam) - def __new__(self, x, y): - pass - check_method(C_new) - - class C_init: - @functools.wraps(mod.spam) - def __init__(self, x, y): - pass - check_method(C_init) + ['x', 'y']) def test_getfullargspec_signature_attr(self): def test(): @@ -970,7 +899,7 @@ def test(): spam_param = inspect.Parameter('spam', inspect.Parameter.POSITIONAL_ONLY) test.__signature__ = inspect.Signature(parameters=(spam_param,)) - self.assertFullArgSpecEquals(test, ['spam'], formatted='(spam)') + self.assertFullArgSpecEquals(test, ['spam']) def test_getfullargspec_signature_annos(self): def test(a:'spam') -> 'ham': pass @@ -984,18 +913,15 @@ def test(): pass @unittest.skipIf(MISSING_C_DOCSTRINGS, "Signature information for builtins requires docstrings") def test_getfullargspec_builtin_methods(self): - self.assertFullArgSpecEquals(_pickle.Pickler.dump, ['self', 'obj'], - formatted='(self, obj)') + self.assertFullArgSpecEquals(_pickle.Pickler.dump, ['self', 'obj']) - self.assertFullArgSpecEquals(_pickle.Pickler(io.BytesIO()).dump, ['self', 'obj'], - formatted='(self, obj)') + self.assertFullArgSpecEquals(_pickle.Pickler(io.BytesIO()).dump, ['self', 'obj']) self.assertFullArgSpecEquals( os.stat, args_e=['path'], kwonlyargs_e=['dir_fd', 'follow_symlinks'], - kwonlydefaults_e={'dir_fd': None, 'follow_symlinks': True}, - formatted='(path, *, dir_fd=None, follow_symlinks=True)') + kwonlydefaults_e={'dir_fd': None, 'follow_symlinks': True}) @cpython_only @unittest.skipIf(MISSING_C_DOCSTRINGS, @@ -1026,12 +952,6 @@ def test_getfullargspec_definition_order_preserved_on_kwonly(self): l = list(signature.kwonlyargs) self.assertEqual(l, unsorted_keyword_only_parameters) - def test_getargspec_method(self): - class A(object): - def m(self): - pass - self.assertArgSpecEquals(A.m, ['self']) - def test_classify_newstyle(self): class A(object): diff --git a/Misc/NEWS.d/next/Library/2021-09-15-10-21-10.bpo-45320.4qaf5x.rst b/Misc/NEWS.d/next/Library/2021-09-15-10-21-10.bpo-45320.4qaf5x.rst new file mode 100644 index 0000000000000..1f2236bd42412 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-15-10-21-10.bpo-45320.4qaf5x.rst @@ -0,0 +1,15 @@ +Removed from the :mod:`inspect` module: + +* the ``getargspec`` function, deprecated since Python 3.0; + use :func:`inspect.signature` or :func:`inspect.getfullargspec` instead. + +* the ``formatargspec`` function, deprecated since Python 3.5; + use the :func:`inspect.signature` function and :class:`Signature` object + directly. + +* the undocumented ``Signature.from_callable`` and ``Signature.from_function`` + functions, deprecated since Python 3.5; use the + :meth:`Signature.from_callable() ` method + instead. + +Patch by Hugo van Kemenade. From webhook-mailer at python.org Wed Oct 20 14:50:14 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 18:50:14 -0000 Subject: [Python-checkins] bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) (GH-29104) Message-ID: https://github.com/python/cpython/commit/b2a989995e6b725c5c957927127832fd3fcecfa2 commit: b2a989995e6b725c5c957927127832fd3fcecfa2 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T20:50:09+02:00 summary: bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) (GH-29104) Co-authored-by: Pablo Galindo Salgado (cherry picked from commit dff0b713436e286bb1afdd7c6f3093c8e8db16dd) Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst M Doc/library/exceptions.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 1a883ba19af4f..f8a692e812791 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -34,6 +34,10 @@ class or one of its subclasses, and not from :exc:`BaseException`. More information on defining exceptions is available in the Python Tutorial under :ref:`tut-userexceptions`. + +Exception context +----------------- + When raising (or re-raising) an exception in an :keyword:`except` or :keyword:`finally` clause :attr:`__context__` is automatically set to the last exception caught; if the @@ -67,6 +71,25 @@ exceptions so that the final line of the traceback always shows the last exception that was raised. +Inheriting from built-in exceptions +----------------------------------- + +User code can create subclasses that inherit from an exception type. +It's recommended to only subclass one exception type at a time to avoid +any possible conflicts between how the bases handle the ``args`` +attribute, as well as due to possible memory layout incompatibilities. + +.. impl-detail:: + + Most built-in exceptions are implemented in C for efficiency, see: + :source:`Objects/exceptions.c`. Some have custom memory layouts + which makes it impossible to create a subclass that inherits from + multiple exception types. The memory layout of a type is an implementation + detail and might change between Python versions, leading to new + conflicts in the future. Therefore, it's recommended to avoid + subclassing multiple exception types altogether. + + Base classes ------------ diff --git a/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst new file mode 100644 index 0000000000000..1721aa2c2dfc4 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst @@ -0,0 +1,4 @@ +Mention in the documentation of :ref:`Built-in Exceptions +` that inheriting from multiple exception types in a +single subclass is not recommended due to possible memory layout +incompatibility. From webhook-mailer at python.org Wed Oct 20 14:50:33 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 18:50:33 -0000 Subject: [Python-checkins] bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) (GH-29105) Message-ID: https://github.com/python/cpython/commit/427ab124b3e9a54602b6f1d073a8e073159c0b51 commit: 427ab124b3e9a54602b6f1d073a8e073159c0b51 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T20:50:28+02:00 summary: bpo-45464: [doc] Explain that subclassing multiple exceptions is fragile (GH-29094) (GH-29105) Co-authored-by: Pablo Galindo Salgado (cherry picked from commit dff0b713436e286bb1afdd7c6f3093c8e8db16dd) Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst M Doc/library/exceptions.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 88ce6847bb392..6c21c6b96d3c6 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -34,6 +34,10 @@ class or one of its subclasses, and not from :exc:`BaseException`. More information on defining exceptions is available in the Python Tutorial under :ref:`tut-userexceptions`. + +Exception context +----------------- + When raising (or re-raising) an exception in an :keyword:`except` or :keyword:`finally` clause :attr:`__context__` is automatically set to the last exception caught; if the @@ -67,6 +71,25 @@ exceptions so that the final line of the traceback always shows the last exception that was raised. +Inheriting from built-in exceptions +----------------------------------- + +User code can create subclasses that inherit from an exception type. +It's recommended to only subclass one exception type at a time to avoid +any possible conflicts between how the bases handle the ``args`` +attribute, as well as due to possible memory layout incompatibilities. + +.. impl-detail:: + + Most built-in exceptions are implemented in C for efficiency, see: + :source:`Objects/exceptions.c`. Some have custom memory layouts + which makes it impossible to create a subclass that inherits from + multiple exception types. The memory layout of a type is an implementation + detail and might change between Python versions, leading to new + conflicts in the future. Therefore, it's recommended to avoid + subclassing multiple exception types altogether. + + Base classes ------------ diff --git a/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst new file mode 100644 index 0000000000000..1721aa2c2dfc4 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-20-16-26-53.bpo-45464.mOISBs.rst @@ -0,0 +1,4 @@ +Mention in the documentation of :ref:`Built-in Exceptions +` that inheriting from multiple exception types in a +single subclass is not recommended due to possible memory layout +incompatibility. From webhook-mailer at python.org Wed Oct 20 14:53:56 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 18:53:56 -0000 Subject: [Python-checkins] bpo-45527: Don't count cache hits, just misses. (GH-29092) Message-ID: https://github.com/python/cpython/commit/bc85eb7a4f16e9e2b6fb713be2466ebb132fd7f2 commit: bc85eb7a4f16e9e2b6fb713be2466ebb132fd7f2 branch: main author: Mark Shannon committer: ambv date: 2021-10-20T20:53:48+02:00 summary: bpo-45527: Don't count cache hits, just misses. (GH-29092) files: M Include/internal/pycore_code.h M Python/ceval.c M Python/specialize.c diff --git a/Include/internal/pycore_code.h b/Include/internal/pycore_code.h index 482bd7eb6ae70..622829fccdd0d 100644 --- a/Include/internal/pycore_code.h +++ b/Include/internal/pycore_code.h @@ -253,53 +253,6 @@ PyAPI_FUNC(PyObject *) _PyCode_GetVarnames(PyCodeObject *); PyAPI_FUNC(PyObject *) _PyCode_GetCellvars(PyCodeObject *); PyAPI_FUNC(PyObject *) _PyCode_GetFreevars(PyCodeObject *); - -/* Cache hits and misses */ - -static inline uint8_t -saturating_increment(uint8_t c) -{ - return c<<1; -} - -static inline uint8_t -saturating_decrement(uint8_t c) -{ - return (c>>1) + 128; -} - -static inline uint8_t -saturating_zero(void) -{ - return 255; -} - -/* Starting value for saturating counter. - * Technically this should be 1, but that is likely to - * cause a bit of thrashing when we optimize then get an immediate miss. - * We want to give the counter a change to stabilize, so we start at 3. - */ -static inline uint8_t -saturating_start(void) -{ - return saturating_zero()<<3; -} - -static inline void -record_cache_hit(_PyAdaptiveEntry *entry) { - entry->counter = saturating_increment(entry->counter); -} - -static inline void -record_cache_miss(_PyAdaptiveEntry *entry) { - entry->counter = saturating_decrement(entry->counter); -} - -static inline int -too_many_cache_misses(_PyAdaptiveEntry *entry) { - return entry->counter == saturating_zero(); -} - #define ADAPTIVE_CACHE_BACKOFF 64 static inline void diff --git a/Python/ceval.c b/Python/ceval.c index f4186dae8a448..adc7b536247b2 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -13,7 +13,7 @@ #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_call.h" // _PyObject_FastCallDictTstate() #include "pycore_ceval.h" // _PyEval_SignalAsyncExc() -#include "pycore_code.h" // saturating_increment() +#include "pycore_code.h" #include "pycore_initconfig.h" // _PyStatus_OK() #include "pycore_long.h" // _PyLong_GetZero() #include "pycore_object.h" // _PyObject_GC_TRACK() @@ -1452,11 +1452,6 @@ eval_frame_handle_pending(PyThreadState *tstate) #define UPDATE_PREV_INSTR_OPARG(instr, oparg) ((uint8_t*)(instr))[-1] = (oparg) -static inline void -record_hit_inline(_Py_CODEUNIT *next_instr, int oparg) -{ - UPDATE_PREV_INSTR_OPARG(next_instr, saturating_increment(oparg)); -} #define GLOBALS() frame->f_globals #define BUILTINS() frame->f_builtins @@ -1480,7 +1475,6 @@ record_hit_inline(_Py_CODEUNIT *next_instr, int oparg) res = ep->me_value; \ DEOPT_IF(res == NULL, LOAD_##attr_or_method); \ STAT_INC(LOAD_##attr_or_method, hit); \ - record_cache_hit(cache0); \ Py_INCREF(res); static int @@ -1976,7 +1970,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyLong_CheckExact(left), BINARY_MULTIPLY); DEOPT_IF(!PyLong_CheckExact(right), BINARY_MULTIPLY); STAT_INC(BINARY_MULTIPLY, hit); - record_hit_inline(next_instr, oparg); PyObject *prod = _PyLong_Multiply((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(prod); Py_DECREF(right); @@ -1994,7 +1987,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyFloat_CheckExact(left), BINARY_MULTIPLY); DEOPT_IF(!PyFloat_CheckExact(right), BINARY_MULTIPLY); STAT_INC(BINARY_MULTIPLY, hit); - record_hit_inline(next_instr, oparg); double dprod = ((PyFloatObject *)left)->ob_fval * ((PyFloatObject *)right)->ob_fval; PyObject *prod = PyFloat_FromDouble(dprod); @@ -2103,7 +2095,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyUnicode_CheckExact(left), BINARY_ADD); DEOPT_IF(Py_TYPE(right) != Py_TYPE(left), BINARY_ADD); STAT_INC(BINARY_ADD, hit); - record_hit_inline(next_instr, oparg); PyObject *res = PyUnicode_Concat(left, right); STACK_SHRINK(1); SET_TOP(res); @@ -2132,7 +2123,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *var = GETLOCAL(next_oparg); DEOPT_IF(var != left, BINARY_ADD); STAT_INC(BINARY_ADD, hit); - record_hit_inline(next_instr, oparg); GETLOCAL(next_oparg) = NULL; Py_DECREF(left); STACK_SHRINK(1); @@ -2150,7 +2140,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyFloat_CheckExact(left), BINARY_ADD); DEOPT_IF(Py_TYPE(right) != Py_TYPE(left), BINARY_ADD); STAT_INC(BINARY_ADD, hit); - record_hit_inline(next_instr, oparg); double dsum = ((PyFloatObject *)left)->ob_fval + ((PyFloatObject *)right)->ob_fval; PyObject *sum = PyFloat_FromDouble(dsum); @@ -2170,7 +2159,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyLong_CheckExact(left), BINARY_ADD); DEOPT_IF(Py_TYPE(right) != Py_TYPE(left), BINARY_ADD); STAT_INC(BINARY_ADD, hit); - record_hit_inline(next_instr, oparg); PyObject *sum = _PyLong_Add((PyLongObject *)left, (PyLongObject *)right); SET_SECOND(sum); Py_DECREF(right); @@ -2241,8 +2229,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(((PyLongObject *)_PyLong_GetZero())->ob_digit[0] == 0); Py_ssize_t index = ((PyLongObject*)sub)->ob_digit[0]; DEOPT_IF(index >= PyList_GET_SIZE(list), BINARY_SUBSCR); - - record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *res = PyList_GET_ITEM(list, index); assert(res != NULL); @@ -2266,8 +2252,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(((PyLongObject *)_PyLong_GetZero())->ob_digit[0] == 0); Py_ssize_t index = ((PyLongObject*)sub)->ob_digit[0]; DEOPT_IF(index >= PyTuple_GET_SIZE(tuple), BINARY_SUBSCR); - - record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *res = PyTuple_GET_ITEM(tuple, index); assert(res != NULL); @@ -2282,7 +2266,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr TARGET(BINARY_SUBSCR_DICT) { PyObject *dict = SECOND(); DEOPT_IF(!PyDict_CheckExact(SECOND()), BINARY_SUBSCR); - record_hit_inline(next_instr, oparg); STAT_INC(BINARY_SUBSCR, hit); PyObject *sub = TOP(); PyObject *res = PyDict_GetItemWithError(dict, sub); @@ -3258,7 +3241,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyDictKeyEntry *ep = DK_ENTRIES(dict->ma_keys) + cache0->index; PyObject *res = ep->me_value; DEOPT_IF(res == NULL, LOAD_GLOBAL); - record_cache_hit(cache0); STAT_INC(LOAD_GLOBAL, hit); Py_INCREF(res); PUSH(res); @@ -3279,7 +3261,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyDictKeyEntry *ep = DK_ENTRIES(bdict->ma_keys) + cache0->index; PyObject *res = ep->me_value; DEOPT_IF(res == NULL, LOAD_GLOBAL); - record_cache_hit(cache0); STAT_INC(LOAD_GLOBAL, hit); Py_INCREF(res); PUSH(res); @@ -3702,7 +3683,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr res = values->values[cache0->index]; DEOPT_IF(res == NULL, LOAD_ATTR); STAT_INC(LOAD_ATTR, hit); - record_cache_hit(cache0); Py_INCREF(res); SET_TOP(res); Py_DECREF(owner); @@ -3742,7 +3722,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr res = ep->me_value; DEOPT_IF(res == NULL, LOAD_ATTR); STAT_INC(LOAD_ATTR, hit); - record_cache_hit(cache0); Py_INCREF(res); SET_TOP(res); Py_DECREF(owner); @@ -3763,7 +3742,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr res = *(PyObject **)addr; DEOPT_IF(res == NULL, LOAD_ATTR); STAT_INC(LOAD_ATTR, hit); - record_cache_hit(cache0); Py_INCREF(res); SET_TOP(res); Py_DECREF(owner); @@ -3805,7 +3783,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyDictValues *values = *(PyDictValues **)(((char *)owner) + tp->tp_inline_values_offset); DEOPT_IF(values == NULL, STORE_ATTR); STAT_INC(STORE_ATTR, hit); - record_cache_hit(cache0); int index = cache0->index; STACK_SHRINK(1); PyObject *value = POP(); @@ -3843,7 +3820,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *old_value = ep->me_value; DEOPT_IF(old_value == NULL, STORE_ATTR); STAT_INC(STORE_ATTR, hit); - record_cache_hit(cache0); STACK_SHRINK(1); PyObject *value = POP(); ep->me_value = value; @@ -3869,7 +3845,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(tp->tp_version_tag != cache1->tp_version, STORE_ATTR); char *addr = (char *)owner + cache0->index; STAT_INC(STORE_ATTR, hit); - record_cache_hit(cache0); STACK_SHRINK(1); PyObject *value = POP(); PyObject *old_value = *(PyObject **)addr; @@ -4527,7 +4502,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *self = TOP(); PyTypeObject *self_cls = Py_TYPE(self); SpecializedCacheEntry *caches = GET_CACHE(); - _PyAdaptiveEntry *cache0 = &caches[0].adaptive; _PyAttrCache *cache1 = &caches[-1].attr; _PyObjectCache *cache2 = &caches[-2].obj; @@ -4538,7 +4512,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(dict != NULL, LOAD_METHOD); DEOPT_IF(((PyHeapTypeObject *)self_cls)->ht_cached_keys->dk_version != cache1->dk_version_or_hint, LOAD_METHOD); STAT_INC(LOAD_METHOD, hit); - record_cache_hit(cache0); PyObject *res = cache2->obj; assert(res != NULL); assert(_PyType_HasFeature(Py_TYPE(res), Py_TPFLAGS_METHOD_DESCRIPTOR)); @@ -4552,13 +4525,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *self = TOP(); PyTypeObject *self_cls = Py_TYPE(self); SpecializedCacheEntry *caches = GET_CACHE(); - _PyAdaptiveEntry *cache0 = &caches[0].adaptive; _PyAttrCache *cache1 = &caches[-1].attr; _PyObjectCache *cache2 = &caches[-2].obj; DEOPT_IF(self_cls->tp_version_tag != cache1->tp_version, LOAD_METHOD); assert(self_cls->tp_dictoffset == 0); STAT_INC(LOAD_METHOD, hit); - record_cache_hit(cache0); PyObject *res = cache2->obj; assert(res != NULL); assert(_PyType_HasFeature(Py_TYPE(res), Py_TPFLAGS_METHOD_DESCRIPTOR)); @@ -4584,7 +4555,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr /* LOAD_METHOD, for class methods */ assert(cframe.use_tracing == 0); SpecializedCacheEntry *caches = GET_CACHE(); - _PyAdaptiveEntry *cache0 = &caches[0].adaptive; _PyAttrCache *cache1 = &caches[-1].attr; _PyObjectCache *cache2 = &caches[-2].obj; @@ -4595,7 +4565,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(cache1->tp_version != 0); STAT_INC(LOAD_METHOD, hit); - record_cache_hit(cache0); PyObject *res = cache2->obj; assert(res != NULL); Py_INCREF(res); @@ -4751,7 +4720,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr /* PEP 523 */ DEOPT_IF(tstate->interp->eval_frame != NULL, CALL_FUNCTION); STAT_INC(CALL_FUNCTION, hit); - record_cache_hit(cache0); InterpreterFrame *new_frame = _PyThreadState_PushFrame( tstate, PyFunction_AS_FRAME_CONSTRUCTOR(func), NULL); if (new_frame == NULL) { @@ -4783,8 +4751,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *callable = SECOND(); DEOPT_IF(!PyCFunction_CheckExact(callable), CALL_FUNCTION); DEOPT_IF(PyCFunction_GET_FLAGS(callable) != METH_O, CALL_FUNCTION); - _PyAdaptiveEntry *cache0 = &GET_CACHE()[0].adaptive; - record_cache_hit(cache0); STAT_INC(CALL_FUNCTION, hit); PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable); @@ -4813,7 +4779,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DEOPT_IF(!PyCFunction_CheckExact(callable), CALL_FUNCTION); DEOPT_IF(PyCFunction_GET_FLAGS(callable) != METH_FASTCALL, CALL_FUNCTION); - record_cache_hit(cache0); STAT_INC(CALL_FUNCTION, hit); PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable); @@ -4845,13 +4810,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(cframe.use_tracing == 0); /* len(o) */ SpecializedCacheEntry *caches = GET_CACHE(); - _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + assert(caches[0].adaptive.original_oparg == 1); _PyObjectCache *cache1 = &caches[-1].obj; - assert(cache0->original_oparg == 1); PyObject *callable = SECOND(); DEOPT_IF(callable != cache1->obj, CALL_FUNCTION); - record_cache_hit(cache0); STAT_INC(CALL_FUNCTION, hit); Py_ssize_t len_i = PyObject_Length(TOP()); @@ -4875,13 +4838,11 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr assert(cframe.use_tracing == 0); /* isinstance(o, o2) */ SpecializedCacheEntry *caches = GET_CACHE(); - _PyAdaptiveEntry *cache0 = &caches[0].adaptive; + assert(caches[0].adaptive.original_oparg == 2); _PyObjectCache *cache1 = &caches[-1].obj; - assert(cache0->original_oparg == 2); PyObject *callable = THIRD(); DEOPT_IF(callable != cache1->obj, CALL_FUNCTION); - record_cache_hit(cache0); STAT_INC(CALL_FUNCTION, hit); int retval = PyObject_IsInstance(SECOND(), TOP()); @@ -5139,8 +5100,8 @@ opname ## _miss: \ { \ STAT_INC(opname, miss); \ _PyAdaptiveEntry *cache = &GET_CACHE()->adaptive; \ - record_cache_miss(cache); \ - if (too_many_cache_misses(cache)) { \ + cache->counter--; \ + if (cache->counter == 0) { \ next_instr[-1] = _Py_MAKECODEUNIT(opname ## _ADAPTIVE, _Py_OPARG(next_instr[-1])); \ STAT_INC(opname, deopt); \ cache_backoff(cache); \ @@ -5154,10 +5115,10 @@ opname ## _miss: \ opname ## _miss: \ { \ STAT_INC(opname, miss); \ - uint8_t oparg = saturating_decrement(_Py_OPARG(next_instr[-1])); \ + uint8_t oparg = _Py_OPARG(next_instr[-1])-1; \ UPDATE_PREV_INSTR_OPARG(next_instr, oparg); \ assert(_Py_OPARG(next_instr[-1]) == oparg); \ - if (oparg == saturating_zero()) /* too many cache misses */ { \ + if (oparg == 0) /* too many cache misses */ { \ oparg = ADAPTIVE_CACHE_BACKOFF; \ next_instr[-1] = _Py_MAKECODEUNIT(opname ## _ADAPTIVE, oparg); \ STAT_INC(opname, deopt); \ diff --git a/Python/specialize.c b/Python/specialize.c index 5cc7082a35a21..162728314e100 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -412,6 +412,18 @@ _Py_Quicken(PyCodeObject *code) { return 0; } +static inline int +initial_counter_value(void) { + /* Starting value for the counter. + * This value needs to be not too low, otherwise + * it would cause excessive de-optimization. + * Neither should it be too high, or that would delay + * de-optimization excessively when it is needed. + * A value around 50 seems to work, and we choose a + * prime number to avoid artifacts. + */ + return 53; +} /* Common */ @@ -770,7 +782,7 @@ _Py_Specialize_LoadAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, Sp success: STAT_INC(LOAD_ATTR, specialization_success); assert(!PyErr_Occurred()); - cache0->counter = saturating_start(); + cache0->counter = initial_counter_value(); return 0; } @@ -852,7 +864,7 @@ _Py_Specialize_StoreAttr(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, S success: STAT_INC(STORE_ATTR, specialization_success); assert(!PyErr_Occurred()); - cache0->counter = saturating_start(); + cache0->counter = initial_counter_value(); return 0; } @@ -1010,7 +1022,7 @@ _Py_Specialize_LoadMethod(PyObject *owner, _Py_CODEUNIT *instr, PyObject *name, success: STAT_INC(LOAD_METHOD, specialization_success); assert(!PyErr_Occurred()); - cache0->counter = saturating_start(); + cache0->counter = initial_counter_value(); return 0; fail: STAT_INC(LOAD_METHOD, specialization_failure); @@ -1086,7 +1098,7 @@ _Py_Specialize_LoadGlobal( success: STAT_INC(LOAD_GLOBAL, specialization_success); assert(!PyErr_Occurred()); - cache0->counter = saturating_start(); + cache0->counter = initial_counter_value(); return 0; } @@ -1137,7 +1149,7 @@ _Py_Specialize_BinarySubscr( PyTypeObject *container_type = Py_TYPE(container); if (container_type == &PyList_Type) { if (PyLong_CheckExact(sub)) { - *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_LIST_INT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_LIST_INT, initial_counter_value()); goto success; } SPECIALIZATION_FAIL(BINARY_SUBSCR, @@ -1146,7 +1158,7 @@ _Py_Specialize_BinarySubscr( } if (container_type == &PyTuple_Type) { if (PyLong_CheckExact(sub)) { - *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_TUPLE_INT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_TUPLE_INT, initial_counter_value()); goto success; } SPECIALIZATION_FAIL(BINARY_SUBSCR, @@ -1154,7 +1166,7 @@ _Py_Specialize_BinarySubscr( goto fail; } if (container_type == &PyDict_Type) { - *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_DICT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_SUBSCR_DICT, initial_counter_value()); goto success; } SPECIALIZATION_FAIL(BINARY_SUBSCR, @@ -1182,19 +1194,19 @@ _Py_Specialize_BinaryAdd(PyObject *left, PyObject *right, _Py_CODEUNIT *instr) if (left_type == &PyUnicode_Type) { int next_opcode = _Py_OPCODE(instr[1]); if (next_opcode == STORE_FAST) { - *instr = _Py_MAKECODEUNIT(BINARY_ADD_UNICODE_INPLACE_FAST, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_ADD_UNICODE_INPLACE_FAST, initial_counter_value()); } else { - *instr = _Py_MAKECODEUNIT(BINARY_ADD_UNICODE, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_ADD_UNICODE, initial_counter_value()); } goto success; } else if (left_type == &PyLong_Type) { - *instr = _Py_MAKECODEUNIT(BINARY_ADD_INT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_ADD_INT, initial_counter_value()); goto success; } else if (left_type == &PyFloat_Type) { - *instr = _Py_MAKECODEUNIT(BINARY_ADD_FLOAT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_ADD_FLOAT, initial_counter_value()); goto success; } @@ -1220,11 +1232,11 @@ _Py_Specialize_BinaryMultiply(PyObject *left, PyObject *right, _Py_CODEUNIT *ins goto fail; } if (PyLong_CheckExact(left)) { - *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_INT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_INT, initial_counter_value()); goto success; } else if (PyFloat_CheckExact(left)) { - *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_FLOAT, saturating_start()); + *instr = _Py_MAKECODEUNIT(BINARY_MULTIPLY_FLOAT, initial_counter_value()); goto success; } else { @@ -1432,7 +1444,7 @@ _Py_Specialize_CallFunction( else { STAT_INC(CALL_FUNCTION, specialization_success); assert(!PyErr_Occurred()); - cache0->counter = saturating_start(); + cache0->counter = initial_counter_value(); } return 0; } From webhook-mailer at python.org Wed Oct 20 15:54:51 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 19:54:51 -0000 Subject: [Python-checkins] bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) Message-ID: https://github.com/python/cpython/commit/6270d3eeaf17b50abc4f8f4d97790d66179638e4 commit: 6270d3eeaf17b50abc4f8f4d97790d66179638e4 branch: main author: Kyungmin Lee committer: ambv date: 2021-10-20T21:54:41+02:00 summary: bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) An object implementing the os.PathLike protocol can represent a file system path as a str or bytes object. Therefore, _infer_return_type function should infer os.PathLike[str] object as str type and os.PathLike[bytes] object as bytes type. files: A Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst M Lib/tempfile.py M Lib/test/test_tempfile.py diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 8570c3ba0627f..531cbf32f1283 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -88,6 +88,10 @@ def _infer_return_type(*args): for arg in args: if arg is None: continue + + if isinstance(arg, _os.PathLike): + arg = _os.fspath(arg) + if isinstance(arg, bytes): if return_type is str: raise TypeError("Can't mix bytes and non-bytes in " diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 96946a281a490..2b0ec46a10327 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -62,6 +62,25 @@ def test_infer_return_type_multiples_and_none(self): def test_infer_return_type_pathlib(self): self.assertIs(str, tempfile._infer_return_type(pathlib.Path('/'))) + def test_infer_return_type_pathlike(self): + class Path: + def __init__(self, path): + self.path = path + + def __fspath__(self): + return self.path + + self.assertIs(str, tempfile._infer_return_type(Path('/'))) + self.assertIs(bytes, tempfile._infer_return_type(Path(b'/'))) + self.assertIs(str, tempfile._infer_return_type('', Path(''))) + self.assertIs(bytes, tempfile._infer_return_type(b'', Path(b''))) + self.assertIs(bytes, tempfile._infer_return_type(None, Path(b''))) + self.assertIs(str, tempfile._infer_return_type(None, Path(''))) + + with self.assertRaises(TypeError): + tempfile._infer_return_type('', Path(b'')) + with self.assertRaises(TypeError): + tempfile._infer_return_type(b'', Path('')) # Common functionality. diff --git a/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst new file mode 100644 index 0000000000000..7dd9795aaa170 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst @@ -0,0 +1,5 @@ +Fix the ``tempfile._infer_return_type`` function so that the ``dir`` +argument of the :mod:`tempfile` functions accepts an object implementing the +``os.PathLike`` protocol. + +Patch by Kyungmin Lee. From webhook-mailer at python.org Wed Oct 20 17:05:34 2021 From: webhook-mailer at python.org (warsaw) Date: Wed, 20 Oct 2021 21:05:34 -0000 Subject: [Python-checkins] bpo-35673: Add a public alias for namespace package __loader__ attribute (#29049) Message-ID: https://github.com/python/cpython/commit/876fc7fcec9a79a11546b7588d3683a5ccb4d31c commit: 876fc7fcec9a79a11546b7588d3683a5ccb4d31c branch: main author: Barry Warsaw committer: warsaw date: 2021-10-20T14:05:29-07:00 summary: bpo-35673: Add a public alias for namespace package __loader__ attribute (#29049) Rename namespace package __loader__ class to be public. Make the old name, i.e. _NamespaceLoader, an alias for the public name, for backward compatibility. files: A Misc/NEWS.d/next/Library/2021-10-18-18-12-47.bpo-35673.KOkHWe.rst M Doc/library/importlib.rst M Lib/importlib/_bootstrap.py M Lib/importlib/_bootstrap_external.py M Lib/importlib/abc.py M Lib/importlib/machinery.py M Lib/test/test_importlib/test_namespace_pkgs.py diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 19230570f2d5f..a25f5145cac83 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -1394,6 +1394,24 @@ find and load modules. .. versionadded:: 3.4 +.. class:: NamespaceLoader(name, path, path_finder): + + A concrete implementation of :class:`importlib.abc.InspectLoader` for + namespace packages. This is an alias for a private class and is only made + public for introspecting the ``__loader__`` attribute on namespace + packages:: + + >>> from importlib.machinery import NamespaceLoader + >>> import my_namespace + >>> isinstance(my_namespace.__loader__, NamespaceLoader) + True + >>> import importlib.abc + >>> isinstance(my_namespace.__loader__, importlib.abc.Loader) + True + + .. versionadded:: 3.11 + + .. class:: ModuleSpec(name, loader, *, origin=None, loader_state=None, is_package=None) A specification for a module's import-system-related state. This is diff --git a/Lib/importlib/_bootstrap.py b/Lib/importlib/_bootstrap.py index 889f08f8aeec1..afb95f4e1df86 100644 --- a/Lib/importlib/_bootstrap.py +++ b/Lib/importlib/_bootstrap.py @@ -507,9 +507,9 @@ def _init_module_attrs(spec, module, *, override=False): if spec.submodule_search_locations is not None: if _bootstrap_external is None: raise NotImplementedError - _NamespaceLoader = _bootstrap_external._NamespaceLoader + NamespaceLoader = _bootstrap_external.NamespaceLoader - loader = _NamespaceLoader.__new__(_NamespaceLoader) + loader = NamespaceLoader.__new__(NamespaceLoader) loader._path = spec.submodule_search_locations spec.loader = loader # While the docs say that module.__file__ is not set for diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index c9692b542a5cc..ef4f23a4b499f 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -1279,8 +1279,10 @@ def append(self, item): self._path.append(item) -# We use this exclusively in module_from_spec() for backward-compatibility. -class _NamespaceLoader: +# This class is actually exposed publicly in a namespace package's __loader__ +# attribute, so it should be available through a non-private name. +# https://bugs.python.org/issue35673 +class NamespaceLoader: def __init__(self, name, path, path_finder): self._path = _NamespacePath(name, path, path_finder) @@ -1291,7 +1293,7 @@ def module_repr(module): The method is deprecated. The import machinery does the job itself. """ - _warnings.warn("_NamespaceLoader.module_repr() is deprecated and " + _warnings.warn("NamespaceLoader.module_repr() is deprecated and " "slated for removal in Python 3.12", DeprecationWarning) return ''.format(module.__name__) @@ -1327,6 +1329,10 @@ def get_resource_reader(self, module): return NamespaceReader(self._path) +# We use this exclusively in module_from_spec() for backward-compatibility. +_NamespaceLoader = NamespaceLoader + + # Finders ##################################################################### class PathFinder: diff --git a/Lib/importlib/abc.py b/Lib/importlib/abc.py index 0b4a3f8071750..1d6843b2ddd44 100644 --- a/Lib/importlib/abc.py +++ b/Lib/importlib/abc.py @@ -213,7 +213,7 @@ def source_to_code(data, path=''): exec_module = _bootstrap_external._LoaderBasics.exec_module load_module = _bootstrap_external._LoaderBasics.load_module -_register(InspectLoader, machinery.BuiltinImporter, machinery.FrozenImporter) +_register(InspectLoader, machinery.BuiltinImporter, machinery.FrozenImporter, machinery.NamespaceLoader) class ExecutionLoader(InspectLoader): diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py index 9a7757fb6e449..d9a19a13f7b27 100644 --- a/Lib/importlib/machinery.py +++ b/Lib/importlib/machinery.py @@ -12,6 +12,7 @@ from ._bootstrap_external import SourceFileLoader from ._bootstrap_external import SourcelessFileLoader from ._bootstrap_external import ExtensionFileLoader +from ._bootstrap_external import NamespaceLoader def all_suffixes(): diff --git a/Lib/test/test_importlib/test_namespace_pkgs.py b/Lib/test/test_importlib/test_namespace_pkgs.py index 3fe3ddc589844..f80283233f977 100644 --- a/Lib/test/test_importlib/test_namespace_pkgs.py +++ b/Lib/test/test_importlib/test_namespace_pkgs.py @@ -1,5 +1,7 @@ import contextlib import importlib +import importlib.abc +import importlib.machinery import os import sys import unittest @@ -342,6 +344,11 @@ def test_path_indexable(self): expected_path = os.path.join(self.root, 'portion1', 'foo') self.assertEqual(foo.__path__[0], expected_path) + def test_loader_abc(self): + import foo + self.assertTrue(isinstance(foo.__loader__, importlib.abc.Loader)) + self.assertTrue(isinstance(foo.__loader__, importlib.machinery.NamespaceLoader)) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-10-18-18-12-47.bpo-35673.KOkHWe.rst b/Misc/NEWS.d/next/Library/2021-10-18-18-12-47.bpo-35673.KOkHWe.rst new file mode 100644 index 0000000000000..e7d6a5fa5a910 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-18-18-12-47.bpo-35673.KOkHWe.rst @@ -0,0 +1,4 @@ +Improve the introspectability of the ``__loader__`` attribute for namespace +packages. :class:`importlib.machinery.NamespaceLoader` is now public, and +implements the :class:`importlib.abc.InspectLoader` interface. ``_NamespaceLoader`` +is kept for backward compatibility. From webhook-mailer at python.org Wed Oct 20 17:25:20 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 21:25:20 -0000 Subject: [Python-checkins] bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) (GH-29112) Message-ID: https://github.com/python/cpython/commit/d33fae7105aaea7c376b5455fd1f8de802ca2542 commit: d33fae7105aaea7c376b5455fd1f8de802ca2542 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T23:25:10+02:00 summary: bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) (GH-29112) An object implementing the os.PathLike protocol can represent a file system path as a str or bytes object. Therefore, _infer_return_type function should infer os.PathLike[str] object as str type and os.PathLike[bytes] object as bytes type. (cherry picked from commit 6270d3eeaf17b50abc4f8f4d97790d66179638e4) Co-authored-by: Kyungmin Lee files: A Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst M Lib/tempfile.py M Lib/test/test_tempfile.py diff --git a/Lib/tempfile.py b/Lib/tempfile.py index 770f72c25295c..eafce6f25b6fb 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -88,6 +88,10 @@ def _infer_return_type(*args): for arg in args: if arg is None: continue + + if isinstance(arg, _os.PathLike): + arg = _os.fspath(arg) + if isinstance(arg, bytes): if return_type is str: raise TypeError("Can't mix bytes and non-bytes in " diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 7dbbf9b7e51e2..8ad1bb98e8e89 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -60,6 +60,25 @@ def test_infer_return_type_multiples_and_none(self): def test_infer_return_type_pathlib(self): self.assertIs(str, tempfile._infer_return_type(pathlib.Path('/'))) + def test_infer_return_type_pathlike(self): + class Path: + def __init__(self, path): + self.path = path + + def __fspath__(self): + return self.path + + self.assertIs(str, tempfile._infer_return_type(Path('/'))) + self.assertIs(bytes, tempfile._infer_return_type(Path(b'/'))) + self.assertIs(str, tempfile._infer_return_type('', Path(''))) + self.assertIs(bytes, tempfile._infer_return_type(b'', Path(b''))) + self.assertIs(bytes, tempfile._infer_return_type(None, Path(b''))) + self.assertIs(str, tempfile._infer_return_type(None, Path(''))) + + with self.assertRaises(TypeError): + tempfile._infer_return_type('', Path(b'')) + with self.assertRaises(TypeError): + tempfile._infer_return_type(b'', Path('')) # Common functionality. diff --git a/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst new file mode 100644 index 0000000000000..7dd9795aaa170 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst @@ -0,0 +1,5 @@ +Fix the ``tempfile._infer_return_type`` function so that the ``dir`` +argument of the :mod:`tempfile` functions accepts an object implementing the +``os.PathLike`` protocol. + +Patch by Kyungmin Lee. From webhook-mailer at python.org Wed Oct 20 17:27:39 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 20 Oct 2021 21:27:39 -0000 Subject: [Python-checkins] bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) (GH-29111) Message-ID: https://github.com/python/cpython/commit/64e83c711eb371d60fce64cae074c4d3311f6ece commit: 64e83c711eb371d60fce64cae074c4d3311f6ece branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-20T23:27:30+02:00 summary: bpo-45192: Fix a bug that infers the type of an os.PathLike[bytes] object as str (GH-28323) (GH-29111) An object implementing the os.PathLike protocol can represent a file system path as a str or bytes object. Therefore, _infer_return_type function should infer os.PathLike[str] object as str type and os.PathLike[bytes] object as bytes type. (cherry picked from commit 6270d3eeaf17b50abc4f8f4d97790d66179638e4) Co-authored-by: Kyungmin Lee files: A Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst M Lib/tempfile.py M Lib/test/test_tempfile.py diff --git a/Lib/tempfile.py b/Lib/tempfile.py index efcf7a7fb3bbc..7b6821240f2b4 100644 --- a/Lib/tempfile.py +++ b/Lib/tempfile.py @@ -88,6 +88,10 @@ def _infer_return_type(*args): for arg in args: if arg is None: continue + + if isinstance(arg, _os.PathLike): + arg = _os.fspath(arg) + if isinstance(arg, bytes): if return_type is str: raise TypeError("Can't mix bytes and non-bytes in " diff --git a/Lib/test/test_tempfile.py b/Lib/test/test_tempfile.py index 96946a281a490..2b0ec46a10327 100644 --- a/Lib/test/test_tempfile.py +++ b/Lib/test/test_tempfile.py @@ -62,6 +62,25 @@ def test_infer_return_type_multiples_and_none(self): def test_infer_return_type_pathlib(self): self.assertIs(str, tempfile._infer_return_type(pathlib.Path('/'))) + def test_infer_return_type_pathlike(self): + class Path: + def __init__(self, path): + self.path = path + + def __fspath__(self): + return self.path + + self.assertIs(str, tempfile._infer_return_type(Path('/'))) + self.assertIs(bytes, tempfile._infer_return_type(Path(b'/'))) + self.assertIs(str, tempfile._infer_return_type('', Path(''))) + self.assertIs(bytes, tempfile._infer_return_type(b'', Path(b''))) + self.assertIs(bytes, tempfile._infer_return_type(None, Path(b''))) + self.assertIs(str, tempfile._infer_return_type(None, Path(''))) + + with self.assertRaises(TypeError): + tempfile._infer_return_type('', Path(b'')) + with self.assertRaises(TypeError): + tempfile._infer_return_type(b'', Path('')) # Common functionality. diff --git a/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst new file mode 100644 index 0000000000000..7dd9795aaa170 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-14-15-52-47.bpo-45192.DjA-BI.rst @@ -0,0 +1,5 @@ +Fix the ``tempfile._infer_return_type`` function so that the ``dir`` +argument of the :mod:`tempfile` functions accepts an object implementing the +``os.PathLike`` protocol. + +Patch by Kyungmin Lee. From webhook-mailer at python.org Wed Oct 20 19:11:51 2021 From: webhook-mailer at python.org (ethanfurman) Date: Wed, 20 Oct 2021 23:11:51 -0000 Subject: [Python-checkins] [3.9] bpo-42517: [ENUM] update docs for changes coming in 3.11 (GH-29113) Message-ID: https://github.com/python/cpython/commit/9733c9651afad84ab2f010e9e68b7c03976ea9f3 commit: 9733c9651afad84ab2f010e9e68b7c03976ea9f3 branch: 3.9 author: Ethan Furman committer: ethanfurman date: 2021-10-20T16:11:47-07:00 summary: [3.9] bpo-42517: [ENUM] update docs for changes coming in 3.11 (GH-29113) files: M Doc/library/enum.rst diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index bbe8bdc82b409..0b8ddc091fe15 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -1125,9 +1125,9 @@ and raise an error if the two do not match:: _Private__names """"""""""""""" -Private names will be normal attributes in Python 3.10 instead of either an error +Private names will be normal attributes in Python 3.11 instead of either an error or a member (depending on if the name ends with an underscore). Using these names -in 3.9 will issue a :exc:`DeprecationWarning`. +in 3.9 and 3.10 will issue a :exc:`DeprecationWarning`. ``Enum`` member type @@ -1150,6 +1150,10 @@ all-uppercase names for members):: >>> FieldTypes.size.value 2 +.. note:: + + This behavior is deprecated and will be removed in 3.11. + .. versionchanged:: 3.5 @@ -1200,3 +1204,8 @@ all named flags and all named combinations of flags that are in the value:: >>> Color(7) # not named combination +.. note:: + + In 3.11 unnamed combinations of flags will only produce the canonical flag + members (aka single-value flags). So ``Color(7)`` would produce something + like ````. From webhook-mailer at python.org Wed Oct 20 20:34:59 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 21 Oct 2021 00:34:59 -0000 Subject: [Python-checkins] bpo-45548: add some missing entries to `Modules/Setup` (GH-29115) Message-ID: https://github.com/python/cpython/commit/dd86f63b551c8c255c6d8082c02f35608d294db9 commit: dd86f63b551c8c255c6d8082c02f35608d294db9 branch: main author: Brett Cannon committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-20T17:34:49-07:00 summary: bpo-45548: add some missing entries to `Modules/Setup` (GH-29115) Also remove a duplicate entry for `_weakref`. files: A Misc/NEWS.d/next/Build/2021-10-20-17-02-56.bpo-45548.BoggEf.rst M Modules/Setup diff --git a/Misc/NEWS.d/next/Build/2021-10-20-17-02-56.bpo-45548.BoggEf.rst b/Misc/NEWS.d/next/Build/2021-10-20-17-02-56.bpo-45548.BoggEf.rst new file mode 100644 index 0000000000000..a52afa21698f9 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-20-17-02-56.bpo-45548.BoggEf.rst @@ -0,0 +1 @@ +Fill in missing entries in Modules/Setup. diff --git a/Modules/Setup b/Modules/Setup index ba8c154ee4586..2e44ec24dacdc 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -171,8 +171,7 @@ _symtable symtablemodule.c #math mathmodule.c _math.c -DPy_BUILD_CORE_MODULE # -lm # math library functions, e.g. sin() #_contextvars _contextvarsmodule.c # Context Variables #_struct -DPy_BUILD_CORE_MODULE _struct.c # binary structure packing/unpacking -#_weakref _weakref.c # basic weak reference support -#_testcapi _testcapimodule.c # Python C API test module +#_testcapi _testcapimodule.c # Python C API test module; CANNOT be statically compiled! #_testinternalcapi _testinternalcapi.c -I$(srcdir)/Include/internal -DPy_BUILD_CORE_MODULE # Python internal C API test module #_random _randommodule.c -DPy_BUILD_CORE_MODULE # Random number generator #_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _elementtree.c # elementtree accelerator @@ -185,6 +184,9 @@ _symtable symtablemodule.c #_json -I$(srcdir)/Include/internal -DPy_BUILD_CORE_BUILTIN _json.c # _json speedups #_statistics _statisticsmodule.c # statistics accelerator #_typing _typingmodule.c # typing accelerator +#_lsprof _lsprof.c rotatingtree.c # cProfile accelerators +#_opcode _opcode.c +#_queue _queuemodule.c -DPy_BUILD_CORE_MODULE #unicodedata unicodedata.c -DPy_BUILD_CORE_BUILTIN # static Unicode character database @@ -197,6 +199,7 @@ _symtable symtablemodule.c #spwd spwdmodule.c # spwd(3) #grp grpmodule.c # grp(3) #select selectmodule.c # select(2); not on ancient System V +#ossaudiodev ossaudiodev.c # Memory-mapped files (also works on Win32). #mmap mmapmodule.c @@ -272,6 +275,10 @@ _symtable symtablemodule.c # _blake module #_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c -DPy_BUILD_CORE_BUILTIN +# Compression +#_bz2 _bz2module.c -lbz2 +#_lzma _lzmamodule.c -llzma + # The _tkinter module. # # The command for _tkinter is long and site specific. Please @@ -373,6 +380,17 @@ _symtable symtablemodule.c # Another example -- the 'xxsubtype' module shows C-level subtyping in action xxsubtype xxsubtype.c +# Limited API examples +#xxlimited xxlimited.c +#xxlimited_35 xxlimited_35.c + +# For testing +#_xxsubinterpreters _xxsubinterpretersmodule.c +#_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c +#_testbuffer _testbuffer.c +#_testimportmultiple _testimportmultiple.c +#_testmultiphase _testmultiphase.c + # Uncommenting the following line tells makesetup that all following modules # are not built (see above for more detail). # From webhook-mailer at python.org Wed Oct 20 22:48:48 2021 From: webhook-mailer at python.org (ethanfurman) Date: Thu, 21 Oct 2021 02:48:48 -0000 Subject: [Python-checkins] bpo-44559: [Enum] restore fixes lost in 3.9 reversion (GH-29114) Message-ID: https://github.com/python/cpython/commit/2a9ab75af32b1ee9f210ae2a0718990687d0f79d commit: 2a9ab75af32b1ee9f210ae2a0718990687d0f79d branch: 3.10 author: Ethan Furman committer: ethanfurman date: 2021-10-20T19:48:37-07:00 summary: bpo-44559: [Enum] restore fixes lost in 3.9 reversion (GH-29114) - fix exception leaks - re-add deprecation warnings files: M Doc/library/enum.rst M Lib/enum.py M Lib/test/test_enum.py diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index e8e49425ab90a..b354a111a3ce7 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -1125,9 +1125,9 @@ and raise an error if the two do not match:: _Private__names """"""""""""""" -Private names will be normal attributes in Python 3.10 instead of either an error +Private names will be normal attributes in Python 3.11 instead of either an error or a member (depending on if the name ends with an underscore). Using these names -in 3.9 will issue a :exc:`DeprecationWarning`. +in 3.10 will issue a :exc:`DeprecationWarning`. ``Enum`` member type @@ -1150,6 +1150,10 @@ all-uppercase names for members):: >>> FieldTypes.size.value 2 +.. note:: + + This behavior is deprecated and will be removed in 3.12. + .. versionchanged:: 3.5 @@ -1200,3 +1204,9 @@ all named flags and all named combinations of flags that are in the value:: >>> Color(7) # not named combination +.. note:: + + In 3.11 unnamed combinations of flags will only produce the canonical flag + members (aka single-value flags). So ``Color(7)`` will produce something + like ````. + diff --git a/Lib/enum.py b/Lib/enum.py index db79e66903ff4..f5657a6eba29c 100644 --- a/Lib/enum.py +++ b/Lib/enum.py @@ -44,10 +44,11 @@ def _is_sunder(name): def _is_private(cls_name, name): # do not use `re` as `re` imports `enum` pattern = '_%s__' % (cls_name, ) + pat_len = len(pattern) if ( - len(name) >= 5 + len(name) > pat_len and name.startswith(pattern) - and name[len(pattern)] != '_' + and name[pat_len:pat_len+1] != ['_'] and (name[-1] != '_' or name[-2] != '_') ): return True @@ -392,12 +393,19 @@ def __call__(cls, value, names=None, *, module=None, qualname=None, type=None, s start=start, ) - def __contains__(cls, member): - if not isinstance(member, Enum): + def __contains__(cls, obj): + if not isinstance(obj, Enum): + import warnings + warnings.warn( + "in 3.12 __contains__ will no longer raise TypeError, but will return True if\n" + "obj is a member or a member's value", + DeprecationWarning, + stacklevel=2, + ) raise TypeError( "unsupported operand type(s) for 'in': '%s' and '%s'" % ( - type(member).__qualname__, cls.__class__.__qualname__)) - return isinstance(member, cls) and member._name_ in cls._member_map_ + type(obj).__qualname__, cls.__class__.__qualname__)) + return isinstance(obj, cls) and obj._name_ in cls._member_map_ def __delattr__(cls, attr): # nicer error message when someone tries to delete an attribute @@ -580,7 +588,7 @@ def _get_mixins_(class_name, bases): return object, Enum def _find_data_type(bases): - data_types = [] + data_types = set() for chain in bases: candidate = None for base in chain.__mro__: @@ -588,19 +596,19 @@ def _find_data_type(bases): continue elif issubclass(base, Enum): if base._member_type_ is not object: - data_types.append(base._member_type_) + data_types.add(base._member_type_) break elif '__new__' in base.__dict__: if issubclass(base, Enum): continue - data_types.append(candidate or base) + data_types.add(candidate or base) break else: candidate = candidate or base if len(data_types) > 1: raise TypeError('%r: too many data types: %r' % (class_name, data_types)) elif data_types: - return data_types[0] + return data_types.pop() else: return None @@ -693,19 +701,25 @@ def __new__(cls, value): except Exception as e: exc = e result = None - if isinstance(result, cls): - return result - else: - ve_exc = ValueError("%r is not a valid %s" % (value, cls.__qualname__)) - if result is None and exc is None: - raise ve_exc - elif exc is None: - exc = TypeError( - 'error in %s._missing_: returned %r instead of None or a valid member' - % (cls.__name__, result) - ) - exc.__context__ = ve_exc - raise exc + try: + if isinstance(result, cls): + return result + else: + ve_exc = ValueError("%r is not a valid %s" % (value, cls.__qualname__)) + if result is None and exc is None: + raise ve_exc + elif exc is None: + exc = TypeError( + 'error in %s._missing_: returned %r instead of None or a valid member' + % (cls.__name__, result) + ) + if not isinstance(exc, ValueError): + exc.__context__ = ve_exc + raise exc + finally: + # ensure all variables that could hold an exception are destroyed + exc = None + ve_exc = None def _generate_next_value_(name, start, count, last_values): """ diff --git a/Lib/test/test_enum.py b/Lib/test/test_enum.py index eb1266b960440..03cf3533fc62d 100644 --- a/Lib/test/test_enum.py +++ b/Lib/test/test_enum.py @@ -11,6 +11,7 @@ from test.support import ALWAYS_EQ, check__all__, threading_helper from datetime import timedelta +python_version = sys.version_info[:2] # for pickle tests try: @@ -347,17 +348,38 @@ class IntLogic(int, Enum): self.assertTrue(IntLogic.true) self.assertFalse(IntLogic.false) - def test_contains(self): + @unittest.skipIf( + python_version >= (3, 12), + '__contains__ now returns True/False for all inputs', + ) + def test_contains_er(self): Season = self.Season self.assertIn(Season.AUTUMN, Season) with self.assertRaises(TypeError): - 3 in Season + with self.assertWarns(DeprecationWarning): + 3 in Season with self.assertRaises(TypeError): - 'AUTUMN' in Season - + with self.assertWarns(DeprecationWarning): + 'AUTUMN' in Season val = Season(3) self.assertIn(val, Season) + # + class OtherEnum(Enum): + one = 1; two = 2 + self.assertNotIn(OtherEnum.two, Season) + @unittest.skipIf( + python_version < (3, 12), + '__contains__ only works with enum memmbers before 3.12', + ) + def test_contains_tf(self): + Season = self.Season + self.assertIn(Season.AUTUMN, Season) + self.assertTrue(3 in Season) + self.assertFalse('AUTUMN' in Season) + val = Season(3) + self.assertIn(val, Season) + # class OtherEnum(Enum): one = 1; two = 2 self.assertNotIn(OtherEnum.two, Season) @@ -1932,6 +1954,38 @@ def _missing_(cls, item): else: raise Exception('Exception not raised.') + def test_missing_exceptions_reset(self): + import weakref + # + class TestEnum(enum.Enum): + VAL1 = 'val1' + VAL2 = 'val2' + # + class Class1: + def __init__(self): + # Gracefully handle an exception of our own making + try: + raise ValueError() + except ValueError: + pass + # + class Class2: + def __init__(self): + # Gracefully handle an exception of Enum's making + try: + TestEnum('invalid_value') + except ValueError: + pass + # No strong refs here so these are free to die. + class_1_ref = weakref.ref(Class1()) + class_2_ref = weakref.ref(Class2()) + # + # The exception raised by Enum creates a reference loop and thus + # Class2 instances will stick around until the next gargage collection + # cycle, unlike Class1. + self.assertIs(class_1_ref(), None) + self.assertIs(class_2_ref(), None) + def test_multiple_mixin(self): class MaxMixin: @classproperty @@ -2085,7 +2139,7 @@ def test_empty_globals(self): exec(code, global_ns, local_ls) @unittest.skipUnless( - sys.version_info[:2] == (3, 9), + python_version == (3, 9), 'private variables are now normal attributes', ) def test_warning_for_private_variables(self): @@ -2390,19 +2444,42 @@ def test_pickle(self): test_pickle_dump_load(self.assertIs, FlagStooges.CURLY|FlagStooges.MOE) test_pickle_dump_load(self.assertIs, FlagStooges) - def test_contains(self): + @unittest.skipIf( + python_version >= (3, 12), + '__contains__ now returns True/False for all inputs', + ) + def test_contains_er(self): Open = self.Open Color = self.Color self.assertFalse(Color.BLACK in Open) self.assertFalse(Open.RO in Color) with self.assertRaises(TypeError): - 'BLACK' in Color + with self.assertWarns(DeprecationWarning): + 'BLACK' in Color with self.assertRaises(TypeError): - 'RO' in Open + with self.assertWarns(DeprecationWarning): + 'RO' in Open with self.assertRaises(TypeError): - 1 in Color + with self.assertWarns(DeprecationWarning): + 1 in Color with self.assertRaises(TypeError): - 1 in Open + with self.assertWarns(DeprecationWarning): + 1 in Open + + @unittest.skipIf( + python_version < (3, 12), + '__contains__ only works with enum memmbers before 3.12', + ) + def test_contains_tf(self): + Open = self.Open + Color = self.Color + self.assertFalse(Color.BLACK in Open) + self.assertFalse(Open.RO in Color) + self.assertFalse('BLACK' in Color) + self.assertFalse('RO' in Open) + self.assertTrue(1 in Color) + self.assertTrue(1 in Open) + def test_member_contains(self): Perm = self.Perm @@ -2883,7 +2960,11 @@ def test_programatic_function_from_empty_tuple(self): self.assertEqual(len(lst), len(Thing)) self.assertEqual(len(Thing), 0, Thing) - def test_contains(self): + @unittest.skipIf( + python_version >= (3, 12), + '__contains__ now returns True/False for all inputs', + ) + def test_contains_er(self): Open = self.Open Color = self.Color self.assertTrue(Color.GREEN in Color) @@ -2891,13 +2972,33 @@ def test_contains(self): self.assertFalse(Color.GREEN in Open) self.assertFalse(Open.RW in Color) with self.assertRaises(TypeError): - 'GREEN' in Color + with self.assertWarns(DeprecationWarning): + 'GREEN' in Color with self.assertRaises(TypeError): - 'RW' in Open + with self.assertWarns(DeprecationWarning): + 'RW' in Open with self.assertRaises(TypeError): - 2 in Color + with self.assertWarns(DeprecationWarning): + 2 in Color with self.assertRaises(TypeError): - 2 in Open + with self.assertWarns(DeprecationWarning): + 2 in Open + + @unittest.skipIf( + python_version < (3, 12), + '__contains__ only works with enum memmbers before 3.12', + ) + def test_contains_tf(self): + Open = self.Open + Color = self.Color + self.assertTrue(Color.GREEN in Color) + self.assertTrue(Open.RW in Open) + self.assertTrue(Color.GREEN in Open) + self.assertTrue(Open.RW in Color) + self.assertFalse('GREEN' in Color) + self.assertFalse('RW' in Open) + self.assertTrue(2 in Color) + self.assertTrue(2 in Open) def test_member_contains(self): Perm = self.Perm @@ -3267,7 +3368,7 @@ def test_convert(self): if name[0:2] not in ('CO', '__')], [], msg='Names other than CONVERT_TEST_* found.') - @unittest.skipUnless(sys.version_info[:2] == (3, 8), + @unittest.skipUnless(python_version == (3, 8), '_convert was deprecated in 3.8') def test_convert_warn(self): with self.assertWarns(DeprecationWarning): @@ -3276,7 +3377,7 @@ def test_convert_warn(self): ('test.test_enum', '__main__')[__name__=='__main__'], filter=lambda x: x.startswith('CONVERT_TEST_')) - @unittest.skipUnless(sys.version_info >= (3, 9), + @unittest.skipUnless(python_version >= (3, 9), '_convert was removed in 3.9') def test_convert_raise(self): with self.assertRaises(AttributeError): @@ -3285,6 +3386,33 @@ def test_convert_raise(self): ('test.test_enum', '__main__')[__name__=='__main__'], filter=lambda x: x.startswith('CONVERT_TEST_')) +class TestHelpers(unittest.TestCase): + + sunder_names = '_bad_', '_good_', '_what_ho_' + dunder_names = '__mal__', '__bien__', '__que_que__' + private_names = '_MyEnum__private', '_MyEnum__still_private' + private_and_sunder_names = '_MyEnum__private_', '_MyEnum__also_private_' + random_names = 'okay', '_semi_private', '_weird__', '_MyEnum__' + + def test_sunder(self): + for name in self.sunder_names + self.private_and_sunder_names: + self.assertTrue(enum._is_sunder(name), '%r is a not sunder name?' % name) + for name in self.dunder_names + self.private_names + self.random_names: + self.assertFalse(enum._is_sunder(name), '%r is a sunder name?' % name) + + def test_dunder(self): + for name in self.dunder_names: + self.assertTrue(enum._is_dunder(name), '%r is a not dunder name?' % name) + for name in self.sunder_names + self.private_names + self.private_and_sunder_names + self.random_names: + self.assertFalse(enum._is_dunder(name), '%r is a dunder name?' % name) + + def test_is_private(self): + for name in self.private_names + self.private_and_sunder_names: + self.assertTrue(enum._is_private('MyEnum', name), '%r is a not private name?') + for name in self.sunder_names + self.dunder_names + self.random_names: + self.assertFalse(enum._is_private('MyEnum', name), '%r is a private name?') + if __name__ == '__main__': unittest.main() + From webhook-mailer at python.org Wed Oct 20 23:32:19 2021 From: webhook-mailer at python.org (ethanfurman) Date: Thu, 21 Oct 2021 03:32:19 -0000 Subject: [Python-checkins] bpo-44174: [Enum] add reference to name mangling (GH-29116) Message-ID: https://github.com/python/cpython/commit/7c4d96103c4e16161e9aed9a584c9857d0674099 commit: 7c4d96103c4e16161e9aed9a584c9857d0674099 branch: main author: Ethan Furman committer: ethanfurman date: 2021-10-20T20:32:11-07:00 summary: bpo-44174: [Enum] add reference to name mangling (GH-29116) files: M Doc/howto/enum.rst diff --git a/Doc/howto/enum.rst b/Doc/howto/enum.rst index 4621b66270fcc..d9cfad972cfa9 100644 --- a/Doc/howto/enum.rst +++ b/Doc/howto/enum.rst @@ -936,7 +936,8 @@ and raise an error if the two do not match:: _Private__names """"""""""""""" -Private names are not converted to enum members, but remain normal attributes. +:ref:`Private names ` are not converted to enum members, +but remain normal attributes. .. versionchanged:: 3.10 From webhook-mailer at python.org Thu Oct 21 04:47:21 2021 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Oct 2021 08:47:21 -0000 Subject: [Python-checkins] bpo-44220: Export PyStructSequence_UnnamedField in the limited API (GH-26331) Message-ID: https://github.com/python/cpython/commit/2cbf50e8126905b57ba9d0d5aa4e238c817d5a03 commit: 2cbf50e8126905b57ba9d0d5aa4e238c817d5a03 branch: main author: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> committer: encukou date: 2021-10-21T10:46:48+02:00 summary: bpo-44220: Export PyStructSequence_UnnamedField in the limited API (GH-26331) files: A Misc/NEWS.d/next/C API/2021-05-24-22-12-40.bpo-44220.H9CUGl.rst M Doc/data/stable_abi.dat M Include/structseq.h M Misc/stable_abi.txt M PC/python3dll.c diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index b41027dee87bc..46ee321b660c3 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -569,6 +569,7 @@ function,PyStructSequence_GetItem,3.2, function,PyStructSequence_New,3.2, function,PyStructSequence_NewType,3.2, function,PyStructSequence_SetItem,3.2, +var,PyStructSequence_UnnamedField,3.11, var,PySuper_Type,3.2, function,PySys_AddWarnOption,3.2, function,PySys_AddWarnOptionUnicode,3.2, diff --git a/Include/structseq.h b/Include/structseq.h index 8f51c89163a4e..e89265a67c322 100644 --- a/Include/structseq.h +++ b/Include/structseq.h @@ -19,7 +19,7 @@ typedef struct PyStructSequence_Desc { int n_in_sequence; } PyStructSequence_Desc; -extern const char * const PyStructSequence_UnnamedField; +PyAPI_DATA(const char * const) PyStructSequence_UnnamedField; #ifndef Py_LIMITED_API PyAPI_FUNC(void) PyStructSequence_InitType(PyTypeObject *type, diff --git a/Misc/NEWS.d/next/C API/2021-05-24-22-12-40.bpo-44220.H9CUGl.rst b/Misc/NEWS.d/next/C API/2021-05-24-22-12-40.bpo-44220.H9CUGl.rst new file mode 100644 index 0000000000000..79ba8aa07cde6 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-05-24-22-12-40.bpo-44220.H9CUGl.rst @@ -0,0 +1 @@ +:c:var:`PyStructSequence_UnnamedField` is added to the Stable ABI. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 8e79f52130622..23e5b96a0e8a7 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -2149,6 +2149,8 @@ function PyType_GetName added 3.11 function PyType_GetQualName added 3.11 +data PyStructSequence_UnnamedField + added 3.11 # (Detailed comments aren't really needed for further entries: from here on # we can use version control logs.) diff --git a/PC/python3dll.c b/PC/python3dll.c index 49b51e69d626e..d9e6fd3e7ca7c 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -842,6 +842,7 @@ EXPORT_DATA(PySeqIter_Type) EXPORT_DATA(PySet_Type) EXPORT_DATA(PySetIter_Type) EXPORT_DATA(PySlice_Type) +EXPORT_DATA(PyStructSequence_UnnamedField) EXPORT_DATA(PySuper_Type) EXPORT_DATA(PyTraceBack_Type) EXPORT_DATA(PyTuple_Type) From webhook-mailer at python.org Thu Oct 21 05:46:31 2021 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Oct 2021 09:46:31 -0000 Subject: [Python-checkins] bpo-45315: PyType_FromSpec: Copy spec->name and have the type own the memory for its name (GH-29103) Message-ID: https://github.com/python/cpython/commit/8a310dd5f4242b5d28013c1905c6573477e3b957 commit: 8a310dd5f4242b5d28013c1905c6573477e3b957 branch: main author: Petr Viktorin committer: encukou date: 2021-10-21T11:46:20+02:00 summary: bpo-45315: PyType_FromSpec: Copy spec->name and have the type own the memory for its name (GH-29103) files: A Misc/NEWS.d/next/C API/2021-10-20-18-41-17.bpo-29103.CMRLyq.rst M Include/cpython/object.h M Lib/test/test_sys.py M Modules/_testcapimodule.c M Objects/typeobject.c diff --git a/Include/cpython/object.h b/Include/cpython/object.h index 849d5aa6a2c72..3a8a256e3b9ae 100644 --- a/Include/cpython/object.h +++ b/Include/cpython/object.h @@ -290,6 +290,7 @@ typedef struct _heaptypeobject { PyObject *ht_name, *ht_slots, *ht_qualname; struct _dictkeysobject *ht_cached_keys; PyObject *ht_module; + char *_ht_tpname; // Storage for "tp_name"; see PyType_FromModuleAndSpec /* here are optional user slots, followed by the members. */ } PyHeapTypeObject; diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 2ce40fcde9eb2..2182898837e50 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -1419,7 +1419,7 @@ def delx(self): del self.__x '3P' # PyMappingMethods '10P' # PySequenceMethods '2P' # PyBufferProcs - '5P') + '6P') class newstyleclass(object): pass # Separate block for PyDictKeysObject with 8 keys and 5 entries check(newstyleclass, s + calcsize(DICT_KEY_STRUCT_FORMAT) + 32 + 21*calcsize("n2P")) diff --git a/Misc/NEWS.d/next/C API/2021-10-20-18-41-17.bpo-29103.CMRLyq.rst b/Misc/NEWS.d/next/C API/2021-10-20-18-41-17.bpo-29103.CMRLyq.rst new file mode 100644 index 0000000000000..e923bfd3ae293 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-20-18-41-17.bpo-29103.CMRLyq.rst @@ -0,0 +1,3 @@ +:c:func:`PyType_FromSpec* ` now copies the class name +from the spec to a buffer owned by the class, so the original can be safely +deallocated. Patch by Petr Viktorin. diff --git a/Modules/_testcapimodule.c b/Modules/_testcapimodule.c index 6857241999ea9..5e4c577962601 100644 --- a/Modules/_testcapimodule.c +++ b/Modules/_testcapimodule.c @@ -1163,6 +1163,126 @@ test_get_type_name(PyObject *self, PyObject *Py_UNUSED(ignored)) } +static PyObject * +simple_str(PyObject *self) { + return PyUnicode_FromString(""); +} + + +static PyObject * +test_type_from_ephemeral_spec(PyObject *self, PyObject *Py_UNUSED(ignored)) +{ + // Test that a heap type can be created from a spec that's later deleted + // (along with all its contents). + // All necessary data must be copied and held by the class + PyType_Spec *spec = NULL; + char *name = NULL; + char *doc = NULL; + PyType_Slot *slots = NULL; + PyObject *class = NULL; + PyObject *instance = NULL; + PyObject *obj = NULL; + PyObject *result = NULL; + + /* create a spec (and all its contents) on the heap */ + + const char NAME[] = "testcapi._Test"; + const char DOC[] = "a test class"; + + spec = PyMem_New(PyType_Spec, 1); + if (spec == NULL) { + PyErr_NoMemory(); + goto finally; + } + name = PyMem_New(char, sizeof(NAME)); + if (name == NULL) { + PyErr_NoMemory(); + goto finally; + } + memcpy(name, NAME, sizeof(NAME)); + + doc = PyMem_New(char, sizeof(DOC)); + if (name == NULL) { + PyErr_NoMemory(); + goto finally; + } + memcpy(doc, DOC, sizeof(DOC)); + + spec->name = name; + spec->basicsize = sizeof(PyObject); + spec->itemsize = 0; + spec->flags = Py_TPFLAGS_DEFAULT; + slots = PyMem_New(PyType_Slot, 3); + if (slots == NULL) { + PyErr_NoMemory(); + goto finally; + } + slots[0].slot = Py_tp_str; + slots[0].pfunc = simple_str; + slots[1].slot = Py_tp_doc; + slots[1].pfunc = doc; + slots[2].slot = 0; + slots[2].pfunc = NULL; + spec->slots = slots; + + /* create the class */ + + class = PyType_FromSpec(spec); + if (class == NULL) { + goto finally; + } + + /* deallocate the spec (and all contents) */ + + // (Explicitly ovewrite memory before freeing, + // so bugs show themselves even without the debug allocator's help.) + memset(spec, 0xdd, sizeof(PyType_Spec)); + PyMem_Del(spec); + spec = NULL; + memset(name, 0xdd, sizeof(NAME)); + PyMem_Del(name); + name = NULL; + memset(doc, 0xdd, sizeof(DOC)); + PyMem_Del(doc); + doc = NULL; + memset(slots, 0xdd, 3 * sizeof(PyType_Slot)); + PyMem_Del(slots); + slots = NULL; + + /* check that everything works */ + + PyTypeObject *class_tp = (PyTypeObject *)class; + PyHeapTypeObject *class_ht = (PyHeapTypeObject *)class; + assert(strcmp(class_tp->tp_name, "testcapi._Test") == 0); + assert(strcmp(PyUnicode_AsUTF8(class_ht->ht_name), "_Test") == 0); + assert(strcmp(PyUnicode_AsUTF8(class_ht->ht_qualname), "_Test") == 0); + assert(strcmp(class_tp->tp_doc, "a test class") == 0); + + // call and check __str__ + instance = PyObject_CallNoArgs(class); + if (instance == NULL) { + goto finally; + } + obj = PyObject_Str(instance); + if (obj == NULL) { + goto finally; + } + assert(strcmp(PyUnicode_AsUTF8(obj), "") == 0); + Py_CLEAR(obj); + + result = Py_NewRef(Py_None); + finally: + PyMem_Del(spec); + PyMem_Del(name); + PyMem_Del(doc); + PyMem_Del(slots); + Py_XDECREF(class); + Py_XDECREF(instance); + Py_XDECREF(obj); + return result; +} + + static PyObject * test_get_type_qualname(PyObject *self, PyObject *Py_UNUSED(ignored)) { @@ -5804,6 +5924,7 @@ static PyMethodDef TestMethods[] = { {"test_get_statictype_slots", test_get_statictype_slots, METH_NOARGS}, {"test_get_type_name", test_get_type_name, METH_NOARGS}, {"test_get_type_qualname", test_get_type_qualname, METH_NOARGS}, + {"test_type_from_ephemeral_spec", test_type_from_ephemeral_spec, METH_NOARGS}, {"get_kwargs", (PyCFunction)(void(*)(void))get_kwargs, METH_VARARGS|METH_KEYWORDS}, {"getargs_tuple", getargs_tuple, METH_VARARGS}, diff --git a/Objects/typeobject.c b/Objects/typeobject.c index aa0733361e58d..18bea476ea9f2 100644 --- a/Objects/typeobject.c +++ b/Objects/typeobject.c @@ -2778,6 +2778,7 @@ type_new_alloc(type_new_ctx *ctx) et->ht_name = Py_NewRef(ctx->name); et->ht_module = NULL; + et->_ht_tpname = NULL; return type; } @@ -3435,25 +3436,42 @@ PyType_FromModuleAndSpec(PyObject *module, PyType_Spec *spec, PyObject *bases) goto fail; } + type = &res->ht_type; + /* The flags must be initialized early, before the GC traverses us */ + type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE; + /* Set the type name and qualname */ const char *s = strrchr(spec->name, '.'); - if (s == NULL) + if (s == NULL) { s = spec->name; - else + } + else { s++; + } - type = &res->ht_type; - /* The flags must be initialized early, before the GC traverses us */ - type->tp_flags = spec->flags | Py_TPFLAGS_HEAPTYPE; res->ht_name = PyUnicode_FromString(s); - if (!res->ht_name) + if (!res->ht_name) { + goto fail; + } + res->ht_qualname = Py_NewRef(res->ht_name); + + /* Copy spec->name to a buffer we own. + * + * Unfortunately, we can't use tp_name directly (with some + * flag saying that it should be deallocated with the type), + * because tp_name is public API and may be set independently + * of any such flag. + * So, we use a separate buffer, _ht_tpname, that's always + * deallocated with the type (if it's non-NULL). + */ + Py_ssize_t name_buf_len = strlen(spec->name) + 1; + res->_ht_tpname = PyMem_Malloc(name_buf_len); + if (res->_ht_tpname == NULL) { goto fail; - res->ht_qualname = res->ht_name; - Py_INCREF(res->ht_qualname); - type->tp_name = spec->name; + } + type->tp_name = memcpy(res->_ht_tpname, spec->name, name_buf_len); - Py_XINCREF(module); - res->ht_module = module; + res->ht_module = Py_XNewRef(module); /* Adjust for empty tuple bases */ if (!bases) { @@ -4082,6 +4100,7 @@ type_dealloc(PyTypeObject *type) _PyDictKeys_DecRef(et->ht_cached_keys); } Py_XDECREF(et->ht_module); + PyMem_Free(et->_ht_tpname); Py_TYPE(type)->tp_free((PyObject *)type); } @@ -4273,10 +4292,12 @@ type_traverse(PyTypeObject *type, visitproc visit, void *arg) Py_VISIT(type->tp_base); Py_VISIT(((PyHeapTypeObject *)type)->ht_module); - /* There's no need to visit type->tp_subclasses or - ((PyHeapTypeObject *)type)->ht_slots, because they can't be involved - in cycles; tp_subclasses is a list of weak references, - and slots is a tuple of strings. */ + /* There's no need to visit others because they can't be involved + in cycles: + type->tp_subclasses is a list of weak references, + ((PyHeapTypeObject *)type)->ht_slots is a tuple of strings, + ((PyHeapTypeObject *)type)->ht_*name are strings. + */ return 0; } From webhook-mailer at python.org Thu Oct 21 06:05:57 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 21 Oct 2021 10:05:57 -0000 Subject: [Python-checkins] bpo-44019: Add operator.call() to __all__ for the operator module (GH-29110) Message-ID: https://github.com/python/cpython/commit/a53456e587c2e935e7e77170d57960e8c80d8a4d commit: a53456e587c2e935e7e77170d57960e8c80d8a4d branch: main author: Kreus Amredes <67752638+Kreusada at users.noreply.github.com> committer: corona10 date: 2021-10-21T19:05:36+09:00 summary: bpo-44019: Add operator.call() to __all__ for the operator module (GH-29110) files: A Misc/NEWS.d/next/Library/2021-10-21-10-14-22.bpo-44019.Xk4Ncr.rst M Lib/operator.py diff --git a/Lib/operator.py b/Lib/operator.py index 72105be05f1c2..30116c1189a49 100644 --- a/Lib/operator.py +++ b/Lib/operator.py @@ -10,7 +10,7 @@ This is the pure Python implementation of the module. """ -__all__ = ['abs', 'add', 'and_', 'attrgetter', 'concat', 'contains', 'countOf', +__all__ = ['abs', 'add', 'and_', 'attrgetter', 'call', 'concat', 'contains', 'countOf', 'delitem', 'eq', 'floordiv', 'ge', 'getitem', 'gt', 'iadd', 'iand', 'iconcat', 'ifloordiv', 'ilshift', 'imatmul', 'imod', 'imul', 'index', 'indexOf', 'inv', 'invert', 'ior', 'ipow', 'irshift', diff --git a/Misc/NEWS.d/next/Library/2021-10-21-10-14-22.bpo-44019.Xk4Ncr.rst b/Misc/NEWS.d/next/Library/2021-10-21-10-14-22.bpo-44019.Xk4Ncr.rst new file mode 100644 index 0000000000000..1793603637fbb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-21-10-14-22.bpo-44019.Xk4Ncr.rst @@ -0,0 +1 @@ +Add :func:`operator.call` to ``operator.__all__``. Patch by Kreusada. From webhook-mailer at python.org Thu Oct 21 06:19:29 2021 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Oct 2021 10:19:29 -0000 Subject: [Python-checkins] bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) Message-ID: https://github.com/python/cpython/commit/5a14f71fe869d4a62dcdeb9a8fbbb5884c75060c commit: 5a14f71fe869d4a62dcdeb9a8fbbb5884c75060c branch: main author: Thomas committer: encukou date: 2021-10-21T12:19:20+02:00 summary: bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) files: M Doc/tutorial/introduction.rst diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 8763626ef553b..33678f5a64b1f 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -11,6 +11,13 @@ with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command. +.. only:: html + + You can toggle the display of prompts and output by clicking on ``>>>`` + in the upper-right corner of an example box. If you hide the prompts + and output for an example, then you can easily copy and paste the input + lines into your interpreter. + .. index:: single: # (hash); comment Many of the examples in this manual, even those entered at the interactive From webhook-mailer at python.org Thu Oct 21 08:41:50 2021 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Oct 2021 12:41:50 -0000 Subject: [Python-checkins] bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) (GH-29119) Message-ID: https://github.com/python/cpython/commit/00ddc1fbd7296ffe066077194a895b175cca26de commit: 00ddc1fbd7296ffe066077194a895b175cca26de branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: encukou date: 2021-10-21T14:41:32+02:00 summary: bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) (GH-29119) (cherry picked from commit 5a14f71fe869d4a62dcdeb9a8fbbb5884c75060c) Co-authored-by: Thomas files: M Doc/tutorial/introduction.rst diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 8763626ef553b..33678f5a64b1f 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -11,6 +11,13 @@ with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command. +.. only:: html + + You can toggle the display of prompts and output by clicking on ``>>>`` + in the upper-right corner of an example box. If you hide the prompts + and output for an example, then you can easily copy and paste the input + lines into your interpreter. + .. index:: single: # (hash); comment Many of the examples in this manual, even those entered at the interactive From webhook-mailer at python.org Thu Oct 21 08:42:15 2021 From: webhook-mailer at python.org (encukou) Date: Thu, 21 Oct 2021 12:42:15 -0000 Subject: [Python-checkins] bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) (GH-29120) Message-ID: https://github.com/python/cpython/commit/bfa4237ecfa605ff94e86fa7141f4a8b1f7cc44a commit: bfa4237ecfa605ff94e86fa7141f4a8b1f7cc44a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: encukou date: 2021-10-21T14:42:10+02:00 summary: bpo-34451: Document prompt and output toggle feature in html tutorial (GH-27105) (GH-29120) (cherry picked from commit 5a14f71fe869d4a62dcdeb9a8fbbb5884c75060c) Co-authored-by: Thomas files: M Doc/tutorial/introduction.rst diff --git a/Doc/tutorial/introduction.rst b/Doc/tutorial/introduction.rst index 8763626ef553b..33678f5a64b1f 100644 --- a/Doc/tutorial/introduction.rst +++ b/Doc/tutorial/introduction.rst @@ -11,6 +11,13 @@ with a prompt are output from the interpreter. Note that a secondary prompt on a line by itself in an example means you must type a blank line; this is used to end a multi-line command. +.. only:: html + + You can toggle the display of prompts and output by clicking on ``>>>`` + in the upper-right corner of an example box. If you hide the prompts + and output for an example, then you can easily copy and paste the input + lines into your interpreter. + .. index:: single: # (hash); comment Many of the examples in this manual, even those entered at the interactive From webhook-mailer at python.org Thu Oct 21 09:12:30 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 21 Oct 2021 13:12:30 -0000 Subject: [Python-checkins] bpo-45522: Allow to disable freelists on build time (GH-29056) Message-ID: https://github.com/python/cpython/commit/9942f42a93ccda047fd3558c47b822e99afe10c0 commit: 9942f42a93ccda047fd3558c47b822e99afe10c0 branch: main author: Christian Heimes committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-21T06:12:20-07:00 summary: bpo-45522: Allow to disable freelists on build time (GH-29056) Freelists for object structs can now be disabled. A new ``configure`` option ``--without-freelists`` can be used to disable all freelists except empty tuple singleton. Internal Py*_MAXFREELIST macros can now be defined as 0 without causing compiler warnings and segfaults. Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/C API/2021-10-19-13-07-46.bpo-45522.kGAwmZ.rst M Doc/whatsnew/3.11.rst M Include/internal/pycore_interp.h M Lib/test/test_sys.py M Objects/dictobject.c M Objects/floatobject.c M Objects/frameobject.c M Objects/genobject.c M Objects/listobject.c M Objects/tupleobject.c M PC/pyconfig.h M Python/context.c M configure M configure.ac M pyconfig.h.in diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index d5e8bc967a816..74fc7536ea231 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -481,6 +481,11 @@ Build Changes ``isinf()``, ``isnan()``, ``round()``. (Contributed by Victor Stinner in :issue:`45440`.) +* Freelists for object structs can now be disabled. A new :program:`configure` + option :option:`!--without-freelists` can be used to disable all freelists + except empty tuple singleton. + (Contributed by Christian Heimes in :issue:`45522`) + C API Changes ============= diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index 0e6edf4ec2670..64ac3abe00fa0 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -85,12 +85,31 @@ struct _Py_unicode_state { struct _Py_unicode_ids ids; }; +#ifndef WITH_FREELISTS +// without freelists +# define PyFloat_MAXFREELIST 0 +// for tuples only store empty tuple singleton +# define PyTuple_MAXSAVESIZE 1 +# define PyTuple_MAXFREELIST 1 +# define PyList_MAXFREELIST 0 +# define PyDict_MAXFREELIST 0 +# define PyFrame_MAXFREELIST 0 +# define _PyAsyncGen_MAXFREELIST 0 +# define PyContext_MAXFREELIST 0 +#endif + +#ifndef PyFloat_MAXFREELIST +# define PyFloat_MAXFREELIST 100 +#endif + struct _Py_float_state { +#if PyFloat_MAXFREELIST > 0 /* Special free list free_list is a singly-linked list of available PyFloatObjects, linked via abuse of their ob_type members. */ int numfree; PyFloatObject *free_list; +#endif }; /* Speed optimization to avoid frequent malloc/free of small tuples */ @@ -119,8 +138,10 @@ struct _Py_tuple_state { #endif struct _Py_list_state { +#if PyList_MAXFREELIST > 0 PyListObject *free_list[PyList_MAXFREELIST]; int numfree; +#endif }; #ifndef PyDict_MAXFREELIST @@ -128,17 +149,25 @@ struct _Py_list_state { #endif struct _Py_dict_state { +#if PyDict_MAXFREELIST > 0 /* Dictionary reuse scheme to save calls to malloc and free */ PyDictObject *free_list[PyDict_MAXFREELIST]; int numfree; PyDictKeysObject *keys_free_list[PyDict_MAXFREELIST]; int keys_numfree; +#endif }; +#ifndef PyFrame_MAXFREELIST +# define PyFrame_MAXFREELIST 200 +#endif + struct _Py_frame_state { +#if PyFrame_MAXFREELIST > 0 PyFrameObject *free_list; /* number of frames currently in free_list */ int numfree; +#endif }; #ifndef _PyAsyncGen_MAXFREELIST @@ -146,6 +175,7 @@ struct _Py_frame_state { #endif struct _Py_async_gen_state { +#if _PyAsyncGen_MAXFREELIST > 0 /* Freelists boost performance 6-10%; they also reduce memory fragmentation, as _PyAsyncGenWrappedValue and PyAsyncGenASend are short-living objects that are instantiated for every @@ -155,12 +185,19 @@ struct _Py_async_gen_state { struct PyAsyncGenASend* asend_freelist[_PyAsyncGen_MAXFREELIST]; int asend_numfree; +#endif }; +#ifndef PyContext_MAXFREELIST +# define PyContext_MAXFREELIST 255 +#endif + struct _Py_context_state { +#if PyContext_MAXFREELIST > 0 // List of free PyContext objects PyContext *freelist; int numfree; +#endif }; struct _Py_exc_state { diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index 2182898837e50..b0688e1e605fe 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -825,7 +825,18 @@ def test_debugmallocstats(self): from test.support.script_helper import assert_python_ok args = ['-c', 'import sys; sys._debugmallocstats()'] ret, out, err = assert_python_ok(*args) - self.assertIn(b"free PyDictObjects", err) + + # Output of sys._debugmallocstats() depends on configure flags. + # The sysconfig vars are not available on Windows. + if sys.platform != "win32": + with_freelists = sysconfig.get_config_var("WITH_FREELISTS") + with_pymalloc = sysconfig.get_config_var("WITH_PYMALLOC") + if with_freelists: + self.assertIn(b"free PyDictObjects", err) + if with_pymalloc: + self.assertIn(b'Small block threshold', err) + if not with_freelists and not with_pymalloc: + self.assertFalse(err) # The function has no parameter self.assertRaises(TypeError, sys._debugmallocstats, True) diff --git a/Misc/NEWS.d/next/C API/2021-10-19-13-07-46.bpo-45522.kGAwmZ.rst b/Misc/NEWS.d/next/C API/2021-10-19-13-07-46.bpo-45522.kGAwmZ.rst new file mode 100644 index 0000000000000..658261f8ce693 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-19-13-07-46.bpo-45522.kGAwmZ.rst @@ -0,0 +1,2 @@ +The internal freelists for frame, float, list, dict, async generators, and +context objects can now be disabled. diff --git a/Objects/dictobject.c b/Objects/dictobject.c index 3d6e4c1e17e1f..6dcd5a1d19715 100644 --- a/Objects/dictobject.c +++ b/Objects/dictobject.c @@ -240,17 +240,20 @@ uint64_t _pydict_global_version = 0; #include "clinic/dictobject.c.h" +#if PyDict_MAXFREELIST > 0 static struct _Py_dict_state * get_dict_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->dict_state; } +#endif void _PyDict_ClearFreeList(PyInterpreterState *interp) { +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = &interp->dict_state; while (state->numfree) { PyDictObject *op = state->free_list[--state->numfree]; @@ -260,6 +263,7 @@ _PyDict_ClearFreeList(PyInterpreterState *interp) while (state->keys_numfree) { PyObject_Free(state->keys_free_list[--state->keys_numfree]); } +#endif } @@ -267,7 +271,7 @@ void _PyDict_Fini(PyInterpreterState *interp) { _PyDict_ClearFreeList(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = &interp->dict_state; state->numfree = -1; state->keys_numfree = -1; @@ -279,9 +283,11 @@ _PyDict_Fini(PyInterpreterState *interp) void _PyDict_DebugMallocStats(FILE *out) { +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = get_dict_state(); _PyDebugAllocatorStats(out, "free PyDictObject", state->numfree, sizeof(PyDictObject)); +#endif } #define DK_MASK(dk) (DK_SIZE(dk)-1) @@ -570,6 +576,7 @@ new_keys_object(uint8_t log2_size) es = sizeof(Py_ssize_t); } +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = get_dict_state(); #ifdef Py_DEBUG // new_keys_object() must not be called after _PyDict_Fini() @@ -579,6 +586,7 @@ new_keys_object(uint8_t log2_size) dk = state->keys_free_list[--state->keys_numfree]; } else +#endif { dk = PyObject_Malloc(sizeof(PyDictKeysObject) + (es< 0 struct _Py_dict_state *state = get_dict_state(); #ifdef Py_DEBUG // free_keys_object() must not be called after _PyDict_Fini() @@ -620,6 +629,7 @@ free_keys_object(PyDictKeysObject *keys) state->keys_free_list[state->keys_numfree++] = keys; return; } +#endif PyObject_Free(keys); } @@ -638,6 +648,7 @@ new_dict(PyDictKeysObject *keys, PyDictValues *values, Py_ssize_t used, int free { PyDictObject *mp; assert(keys != NULL); +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = get_dict_state(); #ifdef Py_DEBUG // new_dict() must not be called after _PyDict_Fini() @@ -649,7 +660,9 @@ new_dict(PyDictKeysObject *keys, PyDictValues *values, Py_ssize_t used, int free assert (Py_IS_TYPE(mp, &PyDict_Type)); _Py_NewReference((PyObject *)mp); } - else { + else +#endif + { mp = PyObject_GC_New(PyDictObject, &PyDict_Type); if (mp == NULL) { dictkeys_decref(keys); @@ -1259,6 +1272,7 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) #ifdef Py_REF_DEBUG _Py_RefTotal--; #endif +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = get_dict_state(); #ifdef Py_DEBUG // dictresize() must not be called after _PyDict_Fini() @@ -1269,7 +1283,9 @@ dictresize(PyDictObject *mp, uint8_t log2_newsize) { state->keys_free_list[state->keys_numfree++] = oldkeys; } - else { + else +#endif + { PyObject_Free(oldkeys); } } @@ -1987,6 +2003,7 @@ dict_dealloc(PyDictObject *mp) assert(keys->dk_refcnt == 1); dictkeys_decref(keys); } +#if PyDict_MAXFREELIST > 0 struct _Py_dict_state *state = get_dict_state(); #ifdef Py_DEBUG // new_dict() must not be called after _PyDict_Fini() @@ -1995,7 +2012,9 @@ dict_dealloc(PyDictObject *mp) if (state->numfree < PyDict_MAXFREELIST && Py_IS_TYPE(mp, &PyDict_Type)) { state->free_list[state->numfree++] = mp; } - else { + else +#endif + { Py_TYPE(mp)->tp_free((PyObject *)mp); } Py_TRASHCAN_END diff --git a/Objects/floatobject.c b/Objects/floatobject.c index 1be31e38d4935..7fc192e720117 100644 --- a/Objects/floatobject.c +++ b/Objects/floatobject.c @@ -28,12 +28,14 @@ class float "PyObject *" "&PyFloat_Type" #endif +#if PyFloat_MAXFREELIST > 0 static struct _Py_float_state * get_float_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->float_state; } +#endif double @@ -126,8 +128,10 @@ PyFloat_GetInfo(void) PyObject * PyFloat_FromDouble(double fval) { + PyFloatObject *op; +#if PyFloat_MAXFREELIST > 0 struct _Py_float_state *state = get_float_state(); - PyFloatObject *op = state->free_list; + op = state->free_list; if (op != NULL) { #ifdef Py_DEBUG // PyFloat_FromDouble() must not be called after _PyFloat_Fini() @@ -136,7 +140,9 @@ PyFloat_FromDouble(double fval) state->free_list = (PyFloatObject *) Py_TYPE(op); state->numfree--; } - else { + else +#endif + { op = PyObject_Malloc(sizeof(PyFloatObject)); if (!op) { return PyErr_NoMemory(); @@ -233,6 +239,7 @@ PyFloat_FromString(PyObject *v) static void float_dealloc(PyFloatObject *op) { +#if PyFloat_MAXFREELIST > 0 if (PyFloat_CheckExact(op)) { struct _Py_float_state *state = get_float_state(); #ifdef Py_DEBUG @@ -247,7 +254,9 @@ float_dealloc(PyFloatObject *op) Py_SET_TYPE(op, (PyTypeObject *)state->free_list); state->free_list = op; } - else { + else +#endif + { Py_TYPE(op)->tp_free((PyObject *)op); } } @@ -2036,6 +2045,7 @@ _PyFloat_InitTypes(void) void _PyFloat_ClearFreeList(PyInterpreterState *interp) { +#if PyFloat_MAXFREELIST > 0 struct _Py_float_state *state = &interp->float_state; PyFloatObject *f = state->free_list; while (f != NULL) { @@ -2045,13 +2055,14 @@ _PyFloat_ClearFreeList(PyInterpreterState *interp) } state->free_list = NULL; state->numfree = 0; +#endif } void _PyFloat_Fini(PyInterpreterState *interp) { _PyFloat_ClearFreeList(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && PyFloat_MAXFREELIST > 0 struct _Py_float_state *state = &interp->float_state; state->numfree = -1; #endif @@ -2061,10 +2072,12 @@ _PyFloat_Fini(PyInterpreterState *interp) void _PyFloat_DebugMallocStats(FILE *out) { +#if PyFloat_MAXFREELIST > 0 struct _Py_float_state *state = get_float_state(); _PyDebugAllocatorStats(out, "free PyFloatObject", state->numfree, sizeof(PyFloatObject)); +#endif } diff --git a/Objects/frameobject.c b/Objects/frameobject.c index 5271790f018af..ffe19b3d99400 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -19,12 +19,14 @@ static PyMemberDef frame_memberlist[] = { {NULL} /* Sentinel */ }; +#if PyFrame_MAXFREELIST > 0 static struct _Py_frame_state * get_frame_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->frame; } +#endif static PyObject * @@ -607,9 +609,6 @@ static PyGetSetDef frame_getsetlist[] = { f_back next item on free list, or NULL */ -/* max value for numfree */ -#define PyFrame_MAXFREELIST 200 - static void _Py_HOT_FUNCTION frame_dealloc(PyFrameObject *f) { @@ -638,6 +637,7 @@ frame_dealloc(PyFrameObject *f) } Py_CLEAR(f->f_back); Py_CLEAR(f->f_trace); +#if PyFrame_MAXFREELIST > 0 struct _Py_frame_state *state = get_frame_state(); #ifdef Py_DEBUG // frame_dealloc() must not be called after _PyFrame_Fini() @@ -648,7 +648,9 @@ frame_dealloc(PyFrameObject *f) f->f_back = state->free_list; state->free_list = f; } - else { + else +#endif + { PyObject_GC_Del(f); } @@ -801,8 +803,10 @@ static inline PyFrameObject* frame_alloc(InterpreterFrame *frame, int owns) { PyFrameObject *f; +#if PyFrame_MAXFREELIST > 0 struct _Py_frame_state *state = get_frame_state(); if (state->free_list == NULL) +#endif { f = PyObject_GC_New(PyFrameObject, &PyFrame_Type); if (f == NULL) { @@ -816,7 +820,9 @@ frame_alloc(InterpreterFrame *frame, int owns) return NULL; } } - else { +#if PyFrame_MAXFREELIST > 0 + else + { #ifdef Py_DEBUG // frame_alloc() must not be called after _PyFrame_Fini() assert(state->numfree != -1); @@ -827,6 +833,7 @@ frame_alloc(InterpreterFrame *frame, int owns) state->free_list = state->free_list->f_back; _Py_NewReference((PyObject *)f); } +#endif f->f_frame = frame; f->f_own_locals_memory = owns; return f; @@ -1069,6 +1076,7 @@ PyFrame_LocalsToFast(PyFrameObject *f, int clear) void _PyFrame_ClearFreeList(PyInterpreterState *interp) { +#if PyFrame_MAXFREELIST > 0 struct _Py_frame_state *state = &interp->frame; while (state->free_list != NULL) { PyFrameObject *f = state->free_list; @@ -1077,13 +1085,14 @@ _PyFrame_ClearFreeList(PyInterpreterState *interp) --state->numfree; } assert(state->numfree == 0); +#endif } void _PyFrame_Fini(PyInterpreterState *interp) { _PyFrame_ClearFreeList(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && PyFrame_MAXFREELIST > 0 struct _Py_frame_state *state = &interp->frame; state->numfree = -1; #endif @@ -1093,10 +1102,12 @@ _PyFrame_Fini(PyInterpreterState *interp) void _PyFrame_DebugMallocStats(FILE *out) { +#if PyFrame_MAXFREELIST > 0 struct _Py_frame_state *state = get_frame_state(); _PyDebugAllocatorStats(out, "free PyFrameObject", state->numfree, sizeof(PyFrameObject)); +#endif } diff --git a/Objects/genobject.c b/Objects/genobject.c index f91f367f9a687..8bf55155eab1c 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -1567,12 +1567,14 @@ PyTypeObject PyAsyncGen_Type = { }; +#if _PyAsyncGen_MAXFREELIST > 0 static struct _Py_async_gen_state * get_async_gen_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->async_gen; } +#endif PyObject * @@ -1595,6 +1597,7 @@ PyAsyncGen_New(PyFrameObject *f, PyObject *name, PyObject *qualname) void _PyAsyncGen_ClearFreeLists(PyInterpreterState *interp) { +#if _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = &interp->async_gen; while (state->value_numfree) { @@ -1610,13 +1613,14 @@ _PyAsyncGen_ClearFreeLists(PyInterpreterState *interp) assert(Py_IS_TYPE(o, &_PyAsyncGenASend_Type)); PyObject_GC_Del(o); } +#endif } void _PyAsyncGen_Fini(PyInterpreterState *interp) { _PyAsyncGen_ClearFreeLists(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = &interp->async_gen; state->value_numfree = -1; state->asend_numfree = -1; @@ -1663,6 +1667,7 @@ async_gen_asend_dealloc(PyAsyncGenASend *o) _PyObject_GC_UNTRACK((PyObject *)o); Py_CLEAR(o->ags_gen); Py_CLEAR(o->ags_sendval); +#if _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = get_async_gen_state(); #ifdef Py_DEBUG // async_gen_asend_dealloc() must not be called after _PyAsyncGen_Fini() @@ -1672,7 +1677,9 @@ async_gen_asend_dealloc(PyAsyncGenASend *o) assert(PyAsyncGenASend_CheckExact(o)); state->asend_freelist[state->asend_numfree++] = o; } - else { + else +#endif + { PyObject_GC_Del(o); } } @@ -1825,6 +1832,7 @@ static PyObject * async_gen_asend_new(PyAsyncGenObject *gen, PyObject *sendval) { PyAsyncGenASend *o; +#if _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = get_async_gen_state(); #ifdef Py_DEBUG // async_gen_asend_new() must not be called after _PyAsyncGen_Fini() @@ -1835,7 +1843,9 @@ async_gen_asend_new(PyAsyncGenObject *gen, PyObject *sendval) o = state->asend_freelist[state->asend_numfree]; _Py_NewReference((PyObject *)o); } - else { + else +#endif + { o = PyObject_GC_New(PyAsyncGenASend, &_PyAsyncGenASend_Type); if (o == NULL) { return NULL; @@ -1863,6 +1873,7 @@ async_gen_wrapped_val_dealloc(_PyAsyncGenWrappedValue *o) { _PyObject_GC_UNTRACK((PyObject *)o); Py_CLEAR(o->agw_val); +#if _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = get_async_gen_state(); #ifdef Py_DEBUG // async_gen_wrapped_val_dealloc() must not be called after _PyAsyncGen_Fini() @@ -1872,7 +1883,9 @@ async_gen_wrapped_val_dealloc(_PyAsyncGenWrappedValue *o) assert(_PyAsyncGenWrappedValue_CheckExact(o)); state->value_freelist[state->value_numfree++] = o; } - else { + else +#endif + { PyObject_GC_Del(o); } } @@ -1936,6 +1949,7 @@ _PyAsyncGenValueWrapperNew(PyObject *val) _PyAsyncGenWrappedValue *o; assert(val); +#if _PyAsyncGen_MAXFREELIST > 0 struct _Py_async_gen_state *state = get_async_gen_state(); #ifdef Py_DEBUG // _PyAsyncGenValueWrapperNew() must not be called after _PyAsyncGen_Fini() @@ -1947,7 +1961,9 @@ _PyAsyncGenValueWrapperNew(PyObject *val) assert(_PyAsyncGenWrappedValue_CheckExact(o)); _Py_NewReference((PyObject*)o); } - else { + else +#endif + { o = PyObject_GC_New(_PyAsyncGenWrappedValue, &_PyAsyncGenWrappedValue_Type); if (o == NULL) { diff --git a/Objects/listobject.c b/Objects/listobject.c index ed5324155f627..e0cae87f45781 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -19,13 +19,14 @@ class list "PyListObject *" "&PyList_Type" #include "clinic/listobject.c.h" - +#if PyList_MAXFREELIST > 0 static struct _Py_list_state * get_list_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->list; } +#endif /* Ensure ob_item has room for at least newsize elements, and set @@ -108,19 +109,21 @@ list_preallocate_exact(PyListObject *self, Py_ssize_t size) void _PyList_ClearFreeList(PyInterpreterState *interp) { +#if PyList_MAXFREELIST > 0 struct _Py_list_state *state = &interp->list; while (state->numfree) { PyListObject *op = state->free_list[--state->numfree]; assert(PyList_CheckExact(op)); PyObject_GC_Del(op); } +#endif } void _PyList_Fini(PyInterpreterState *interp) { _PyList_ClearFreeList(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && PyList_MAXFREELIST > 0 struct _Py_list_state *state = &interp->list; state->numfree = -1; #endif @@ -130,32 +133,38 @@ _PyList_Fini(PyInterpreterState *interp) void _PyList_DebugMallocStats(FILE *out) { +#if PyList_MAXFREELIST > 0 struct _Py_list_state *state = get_list_state(); _PyDebugAllocatorStats(out, "free PyListObject", state->numfree, sizeof(PyListObject)); +#endif } PyObject * PyList_New(Py_ssize_t size) { + PyListObject *op; + if (size < 0) { PyErr_BadInternalCall(); return NULL; } +#if PyList_MAXFREELIST > 0 struct _Py_list_state *state = get_list_state(); - PyListObject *op; #ifdef Py_DEBUG // PyList_New() must not be called after _PyList_Fini() assert(state->numfree != -1); #endif - if (state->numfree) { + if (PyList_MAXFREELIST && state->numfree) { state->numfree--; op = state->free_list[state->numfree]; _Py_NewReference((PyObject *)op); } - else { + else +#endif + { op = PyObject_GC_New(PyListObject, &PyList_Type); if (op == NULL) { return NULL; @@ -344,6 +353,7 @@ list_dealloc(PyListObject *op) } PyMem_Free(op->ob_item); } +#if PyList_MAXFREELIST > 0 struct _Py_list_state *state = get_list_state(); #ifdef Py_DEBUG // list_dealloc() must not be called after _PyList_Fini() @@ -352,7 +362,9 @@ list_dealloc(PyListObject *op) if (state->numfree < PyList_MAXFREELIST && PyList_CheckExact(op)) { state->free_list[state->numfree++] = op; } - else { + else +#endif + { Py_TYPE(op)->tp_free((PyObject *)op); } Py_TRASHCAN_END diff --git a/Objects/tupleobject.c b/Objects/tupleobject.c index 051683086ea2c..e9d1b5926abb3 100644 --- a/Objects/tupleobject.c +++ b/Objects/tupleobject.c @@ -66,7 +66,8 @@ tuple_alloc(Py_ssize_t size) return NULL; } -#if PyTuple_MAXSAVESIZE > 0 +// Check for max save size > 1. Empty tuple singleton is special case. +#if PyTuple_MAXSAVESIZE > 1 struct _Py_tuple_state *state = get_tuple_state(); #ifdef Py_DEBUG // tuple_alloc() must not be called after _PyTuple_Fini() diff --git a/PC/pyconfig.h b/PC/pyconfig.h index bb55ff4fe14c4..b3e73d4dabefe 100644 --- a/PC/pyconfig.h +++ b/PC/pyconfig.h @@ -463,6 +463,9 @@ Py_NO_ENABLE_SHARED to find out. Also support MS_NO_COREDLL for b/w compat */ /* Use Python's own small-block memory-allocator. */ #define WITH_PYMALLOC 1 +/* Define if you want to compile in object freelists optimization */ +#define WITH_FREELISTS 1 + /* Define if you have clock. */ /* #define HAVE_CLOCK */ diff --git a/Python/context.c b/Python/context.c index d78f7f993bb89..a20ec7123731c 100644 --- a/Python/context.c +++ b/Python/context.c @@ -9,9 +9,6 @@ #include "structmember.h" // PyMemberDef -#define CONTEXT_FREELIST_MAXLEN 255 - - #include "clinic/context.c.h" /*[clinic input] module _contextvars @@ -66,12 +63,14 @@ static int contextvar_del(PyContextVar *var); +#if PyContext_MAXFREELIST > 0 static struct _Py_context_state * get_context_state(void) { PyInterpreterState *interp = _PyInterpreterState_GET(); return &interp->context; } +#endif PyObject * @@ -340,8 +339,9 @@ class _contextvars.Context "PyContext *" "&PyContext_Type" static inline PyContext * _context_alloc(void) { - struct _Py_context_state *state = get_context_state(); PyContext *ctx; +#if PyContext_MAXFREELIST > 0 + struct _Py_context_state *state = get_context_state(); #ifdef Py_DEBUG // _context_alloc() must not be called after _PyContext_Fini() assert(state->numfree != -1); @@ -353,7 +353,9 @@ _context_alloc(void) ctx->ctx_weakreflist = NULL; _Py_NewReference((PyObject *)ctx); } - else { + else +#endif + { ctx = PyObject_GC_New(PyContext, &PyContext_Type); if (ctx == NULL) { return NULL; @@ -469,17 +471,20 @@ context_tp_dealloc(PyContext *self) } (void)context_tp_clear(self); +#if PyContext_MAXFREELIST > 0 struct _Py_context_state *state = get_context_state(); #ifdef Py_DEBUG // _context_alloc() must not be called after _PyContext_Fini() assert(state->numfree != -1); #endif - if (state->numfree < CONTEXT_FREELIST_MAXLEN) { + if (state->numfree < PyContext_MAXFREELIST) { state->numfree++; self->ctx_weakreflist = (PyObject *)state->freelist; state->freelist = self; } - else { + else +#endif + { Py_TYPE(self)->tp_free(self); } } @@ -1289,6 +1294,7 @@ get_token_missing(void) void _PyContext_ClearFreeList(PyInterpreterState *interp) { +#if PyContext_MAXFREELIST > 0 struct _Py_context_state *state = &interp->context; for (; state->numfree; state->numfree--) { PyContext *ctx = state->freelist; @@ -1296,6 +1302,7 @@ _PyContext_ClearFreeList(PyInterpreterState *interp) ctx->ctx_weakreflist = NULL; PyObject_GC_Del(ctx); } +#endif } @@ -1306,7 +1313,7 @@ _PyContext_Fini(PyInterpreterState *interp) Py_CLEAR(_token_missing); } _PyContext_ClearFreeList(interp); -#ifdef Py_DEBUG +#if defined(Py_DEBUG) && PyContext_MAXFREELIST > 0 struct _Py_context_state *state = &interp->context; state->numfree = -1; #endif diff --git a/configure b/configure index ec7c72c7f3276..198b0703fd6af 100755 --- a/configure +++ b/configure @@ -846,6 +846,7 @@ with_dbmliborder enable_ipv6 with_doc_strings with_pymalloc +with_freelists with_c_locale_coercion with_valgrind with_dtrace @@ -1588,6 +1589,7 @@ Optional Packages: names `ndbm', `gdbm' and `bdb'. --with-doc-strings enable documentation strings (default is yes) --with-pymalloc enable specialized mallocs (default is yes) + --with-freelists enable object freelists (default is yes) --with-c-locale-coercion enable C locale coercion to a UTF-8 based locale (default is yes) @@ -11728,6 +11730,30 @@ fi { $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_pymalloc" >&5 $as_echo "$with_pymalloc" >&6; } +# Check whether objects such as float, tuple and dict are using +# freelists to optimization memory allocation. +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-freelists" >&5 +$as_echo_n "checking for --with-freelists... " >&6; } + +# Check whether --with-freelists was given. +if test "${with_freelists+set}" = set; then : + withval=$with_freelists; +fi + + +if test -z "$with_freelists" +then + with_freelists="yes" +fi +if test "$with_freelists" != "no" +then + +$as_echo "#define WITH_FREELISTS 1" >>confdefs.h + +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_freelists" >&5 +$as_echo "$with_freelists" >&6; } + # Check for --with-c-locale-coercion { $as_echo "$as_me:${as_lineno-$LINENO}: checking for --with-c-locale-coercion" >&5 $as_echo_n "checking for --with-c-locale-coercion... " >&6; } diff --git a/configure.ac b/configure.ac index c0259524756f2..edda08daac117 100644 --- a/configure.ac +++ b/configure.ac @@ -3616,6 +3616,23 @@ then fi AC_MSG_RESULT($with_pymalloc) +# Check whether objects such as float, tuple and dict are using +# freelists to optimization memory allocation. +AC_MSG_CHECKING(for --with-freelists) +AC_ARG_WITH(freelists, + AS_HELP_STRING([--with-freelists], [enable object freelists (default is yes)])) + +if test -z "$with_freelists" +then + with_freelists="yes" +fi +if test "$with_freelists" != "no" +then + AC_DEFINE(WITH_FREELISTS, 1, + [Define if you want to compile in object freelists optimization]) +fi +AC_MSG_RESULT($with_freelists) + # Check for --with-c-locale-coercion AC_MSG_CHECKING(for --with-c-locale-coercion) AC_ARG_WITH(c-locale-coercion, diff --git a/pyconfig.h.in b/pyconfig.h.in index a426e8effddb9..081ea61bae834 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -1555,6 +1555,9 @@ /* Define to build the readline module against Editline. */ #undef WITH_EDITLINE +/* Define if you want to compile in object freelists optimization */ +#undef WITH_FREELISTS + /* Define to 1 if libintl is needed for locale functions. */ #undef WITH_LIBINTL From webhook-mailer at python.org Thu Oct 21 09:38:41 2021 From: webhook-mailer at python.org (nascheme) Date: Thu, 21 Oct 2021 13:38:41 -0000 Subject: [Python-checkins] bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) Message-ID: https://github.com/python/cpython/commit/311910b31a4bd94dc79298388b7cb65ca5546438 commit: 311910b31a4bd94dc79298388b7cb65ca5546438 branch: main author: Neil Schemenauer committer: nascheme date: 2021-10-21T06:38:36-07:00 summary: bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) MAP_BOT_LENGTH was incorrectly used to compute MAP_TOP_MASK instead of MAP_TOP_LENGTH. On 64-bit machines, the error causes the tree to hold 46-bits of virtual addresses, rather than the intended 48-bits. files: A Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst M Objects/obmalloc.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst new file mode 100644 index 0000000000000..3a082a4ffdbcb --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst @@ -0,0 +1,3 @@ +Fix a bug in the obmalloc radix tree code. On 64-bit machines, the bug +causes the tree to hold 46-bits of virtual addresses, rather than the +intended 48-bits. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index d8d6f6dea0d53..2eddb2cafd498 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -1341,7 +1341,7 @@ _Py_GetAllocatedBlocks(void) #define MAP_TOP_BITS INTERIOR_BITS #define MAP_TOP_LENGTH (1 << MAP_TOP_BITS) -#define MAP_TOP_MASK (MAP_BOT_LENGTH - 1) +#define MAP_TOP_MASK (MAP_TOP_LENGTH - 1) #define MAP_MID_BITS INTERIOR_BITS #define MAP_MID_LENGTH (1 << MAP_MID_BITS) From webhook-mailer at python.org Thu Oct 21 11:40:47 2021 From: webhook-mailer at python.org (nascheme) Date: Thu, 21 Oct 2021 15:40:47 -0000 Subject: [Python-checkins] bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) (GH-29122) Message-ID: https://github.com/python/cpython/commit/1cdac61065e72db60d26e03ef9286d2743d7000e commit: 1cdac61065e72db60d26e03ef9286d2743d7000e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: nascheme date: 2021-10-21T08:39:58-07:00 summary: bpo-45521: Fix a bug in the obmalloc radix tree code. (GH-29051) (GH-29122) MAP_BOT_LENGTH was incorrectly used to compute MAP_TOP_MASK instead of MAP_TOP_LENGTH. On 64-bit machines, the error causes the tree to hold 46-bits of virtual addresses, rather than the intended 48-bits. (cherry picked from commit 311910b31a4bd94dc79298388b7cb65ca5546438) files: A Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst M Objects/obmalloc.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst new file mode 100644 index 0000000000000..3a082a4ffdbcb --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-18-22-40-33.bpo-45521.GdMiuW.rst @@ -0,0 +1,3 @@ +Fix a bug in the obmalloc radix tree code. On 64-bit machines, the bug +causes the tree to hold 46-bits of virtual addresses, rather than the +intended 48-bits. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 15c442b858d5f..615703a963ede 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -1328,7 +1328,7 @@ _Py_GetAllocatedBlocks(void) #define MAP_TOP_BITS INTERIOR_BITS #define MAP_TOP_LENGTH (1 << MAP_TOP_BITS) -#define MAP_TOP_MASK (MAP_BOT_LENGTH - 1) +#define MAP_TOP_MASK (MAP_TOP_LENGTH - 1) #define MAP_MID_BITS INTERIOR_BITS #define MAP_MID_LENGTH (1 << MAP_MID_BITS) From webhook-mailer at python.org Thu Oct 21 12:50:42 2021 From: webhook-mailer at python.org (ethanfurman) Date: Thu, 21 Oct 2021 16:50:42 -0000 Subject: [Python-checkins] [3.10] bpo-44174: [Enum] add reference to name mangling (GH-29117) Message-ID: https://github.com/python/cpython/commit/828722aca4ccba893f6b2e8c1d41fd74fd6e208d commit: 828722aca4ccba893f6b2e8c1d41fd74fd6e208d branch: 3.10 author: Ethan Furman committer: ethanfurman date: 2021-10-21T09:50:29-07:00 summary: [3.10] bpo-44174: [Enum] add reference to name mangling (GH-29117) files: M Doc/library/enum.rst diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index b354a111a3ce73..eb5ffd0c5d51c5 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -1125,9 +1125,9 @@ and raise an error if the two do not match:: _Private__names """"""""""""""" -Private names will be normal attributes in Python 3.11 instead of either an error -or a member (depending on if the name ends with an underscore). Using these names -in 3.10 will issue a :exc:`DeprecationWarning`. +:ref:`Private names ` will be normal attributes in Python +3.11 instead of either an error or a member (depending on if the name ends with +an underscore). Using these names in 3.10 will issue a :exc:`DeprecationWarning`. ``Enum`` member type From webhook-mailer at python.org Thu Oct 21 14:32:33 2021 From: webhook-mailer at python.org (ethanfurman) Date: Thu, 21 Oct 2021 18:32:33 -0000 Subject: [Python-checkins] [3.9] bpo-44174: [Enum] add name-mangling reference (GH-29128) Message-ID: https://github.com/python/cpython/commit/e628700dbf2c3376502cbb5a9bff2d58d1102e16 commit: e628700dbf2c3376502cbb5a9bff2d58d1102e16 branch: 3.9 author: Ethan Furman committer: ethanfurman date: 2021-10-21T11:32:18-07:00 summary: [3.9] bpo-44174: [Enum] add name-mangling reference (GH-29128) files: M Doc/library/enum.rst diff --git a/Doc/library/enum.rst b/Doc/library/enum.rst index 0b8ddc091fe15..4e15901c66bfc 100644 --- a/Doc/library/enum.rst +++ b/Doc/library/enum.rst @@ -1125,7 +1125,7 @@ and raise an error if the two do not match:: _Private__names """"""""""""""" -Private names will be normal attributes in Python 3.11 instead of either an error +:ref:`Private names ` will be normal attributes in Python 3.11 instead of either an error or a member (depending on if the name ends with an underscore). Using these names in 3.9 and 3.10 will issue a :exc:`DeprecationWarning`. @@ -1152,7 +1152,7 @@ all-uppercase names for members):: .. note:: - This behavior is deprecated and will be removed in 3.11. + This behavior is deprecated and will be removed in 3.12. .. versionchanged:: 3.5 From webhook-mailer at python.org Thu Oct 21 16:16:59 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 20:16:59 -0000 Subject: [Python-checkins] Move several typing tests to a proper class, refs GH-28563 (GH-29126) Message-ID: https://github.com/python/cpython/commit/0c4c2e6213f348dc98a787e385cde0a480e80ee9 commit: 0c4c2e6213f348dc98a787e385cde0a480e80ee9 branch: main author: Nikita Sobolev committer: ambv date: 2021-10-21T22:16:50+02:00 summary: Move several typing tests to a proper class, refs GH-28563 (GH-29126) files: M Lib/test/test_typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 032fe91c7a840..b1414dc82bae0 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -3282,6 +3282,22 @@ class BadType(BadBase): self.assertNotIn('bad', sys.modules) self.assertEqual(get_type_hints(BadType), {'foo': tuple, 'bar': list}) + def test_forward_ref_and_final(self): + # https://bugs.python.org/issue45166 + hints = get_type_hints(ann_module5) + self.assertEqual(hints, {'name': Final[str]}) + + hints = get_type_hints(ann_module5.MyClass) + self.assertEqual(hints, {'value': Final}) + + def test_top_level_class_var(self): + # https://bugs.python.org/issue45166 + with self.assertRaisesRegex( + TypeError, + r'typing.ClassVar\[int\] is not valid as type argument', + ): + get_type_hints(ann_module6) + class GetUtilitiesTestCase(TestCase): def test_get_origin(self): @@ -3345,22 +3361,6 @@ class C(Generic[T]): pass (Concatenate[int, P], int)) self.assertEqual(get_args(list | str), (list, str)) - def test_forward_ref_and_final(self): - # https://bugs.python.org/issue45166 - hints = get_type_hints(ann_module5) - self.assertEqual(hints, {'name': Final[str]}) - - hints = get_type_hints(ann_module5.MyClass) - self.assertEqual(hints, {'value': Final}) - - def test_top_level_class_var(self): - # https://bugs.python.org/issue45166 - with self.assertRaisesRegex( - TypeError, - r'typing.ClassVar\[int\] is not valid as type argument', - ): - get_type_hints(ann_module6) - class CollectionsAbcTests(BaseTestCase): From webhook-mailer at python.org Thu Oct 21 16:26:02 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 20:26:02 -0000 Subject: [Python-checkins] bpo-45160: Ttk optionmenu only set variable once (GH-28291) Message-ID: https://github.com/python/cpython/commit/add46f84769a7e6fafa50954f79b7c248231fa4e commit: add46f84769a7e6fafa50954f79b7c248231fa4e branch: main author: E-Paine <63801254+E-Paine at users.noreply.github.com> committer: ambv date: 2021-10-21T22:25:52+02:00 summary: bpo-45160: Ttk optionmenu only set variable once (GH-28291) files: A Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/ttk.py diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index 438d21d0b3733..cddd1f2e84834 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -301,6 +301,19 @@ def test_unique_radiobuttons(self): optmenu.destroy() optmenu2.destroy() + def test_trace_variable(self): + # prior to bpo45160, tracing a variable would cause the callback to be made twice + success = [] + items = ('a', 'b', 'c') + textvar = tkinter.StringVar(self.root) + def cb_test(*args): + self.assertEqual(textvar.get(), items[1]) + success.append(True) + optmenu = ttk.OptionMenu(self.root, textvar, "a", *items) + textvar.trace("w", cb_test) + optmenu['menu'].invoke(1) + self.assertEqual(success, [True]) + class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index b854235a62679..acdd565ec48a9 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -1643,7 +1643,10 @@ def set_menu(self, default=None, *values): menu.delete(0, 'end') for val in values: menu.add_radiobutton(label=val, - command=tkinter._setit(self._variable, val, self._callback), + command=( + None if self._callback is None + else lambda val=val: self._callback(val) + ), variable=self._variable) if default: diff --git a/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst new file mode 100644 index 0000000000000..9d11ed0e55d24 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst @@ -0,0 +1 @@ +When tracing a tkinter variable used by a ttk OptionMenu, callbacks are no longer made twice. \ No newline at end of file From webhook-mailer at python.org Thu Oct 21 16:34:27 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 20:34:27 -0000 Subject: [Python-checkins] Add workflow_dispatch trigger to GHA workflows (GH-27873) Message-ID: https://github.com/python/cpython/commit/3754f55b36206091eda057202666f7905ab3d0e3 commit: 3754f55b36206091eda057202666f7905ab3d0e3 branch: main author: Ryan Mast <3969255+nightlark at users.noreply.github.com> committer: ambv date: 2021-10-21T22:34:18+02:00 summary: Add workflow_dispatch trigger to GHA workflows (GH-27873) files: M .github/workflows/build.yml M .github/workflows/build_msi.yml M .github/workflows/doc.yml diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml index ec3acf77563f7..a11a368fd8311 100644 --- a/.github/workflows/build.yml +++ b/.github/workflows/build.yml @@ -4,6 +4,7 @@ name: Tests # it prevents to mark a job as mandatory. A PR cannot be merged if a job is # mandatory but not scheduled because of "paths-ignore". on: + workflow_dispatch: push: branches: - 'main' diff --git a/.github/workflows/build_msi.yml b/.github/workflows/build_msi.yml index db5eaa17573f2..476a0b1bc0738 100644 --- a/.github/workflows/build_msi.yml +++ b/.github/workflows/build_msi.yml @@ -1,6 +1,7 @@ name: TestsMSI on: + workflow_dispatch: push: branches: - 'main' diff --git a/.github/workflows/doc.yml b/.github/workflows/doc.yml index ca4476c509012..755a4d5e173ee 100644 --- a/.github/workflows/doc.yml +++ b/.github/workflows/doc.yml @@ -1,6 +1,7 @@ name: Docs on: + workflow_dispatch: #push: # branches: # - 'main' From webhook-mailer at python.org Thu Oct 21 16:43:05 2021 From: webhook-mailer at python.org (ericvsmith) Date: Thu, 21 Oct 2021 20:43:05 -0000 Subject: [Python-checkins] bpo-45557: Fix underscore_numbers in pprint.pprint(). (GH-29129) Message-ID: https://github.com/python/cpython/commit/087f089e5e04d5b132ffbff0576667d591f13219 commit: 087f089e5e04d5b132ffbff0576667d591f13219 branch: main author: Eric V. Smith committer: ericvsmith date: 2021-10-21T16:42:55-04:00 summary: bpo-45557: Fix underscore_numbers in pprint.pprint(). (GH-29129) files: A Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst M Lib/pprint.py diff --git a/Lib/pprint.py b/Lib/pprint.py index 60ce57e910ec4..575688d8eb6f4 100644 --- a/Lib/pprint.py +++ b/Lib/pprint.py @@ -50,7 +50,8 @@ def pprint(object, stream=None, indent=1, width=80, depth=None, *, """Pretty-print a Python object to a stream [default is sys.stdout].""" printer = PrettyPrinter( stream=stream, indent=indent, width=width, depth=depth, - compact=compact, sort_dicts=sort_dicts, underscore_numbers=False) + compact=compact, sort_dicts=sort_dicts, + underscore_numbers=underscore_numbers) printer.pprint(object) def pformat(object, indent=1, width=80, depth=None, *, diff --git a/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst b/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst new file mode 100644 index 0000000000000..7472b08d1b08e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst @@ -0,0 +1,2 @@ +pprint.pprint() now handles underscore_numbers correctly. Previously it was +always setting it to False. From webhook-mailer at python.org Thu Oct 21 16:57:58 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 20:57:58 -0000 Subject: [Python-checkins] bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) Message-ID: https://github.com/python/cpython/commit/887a55705bb6c05a507c2886c9978a9e0cff0dd7 commit: 887a55705bb6c05a507c2886c9978a9e0cff0dd7 branch: main author: Mark Dickinson committer: ambv date: 2021-10-21T22:57:49+02:00 summary: bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) Co-authored-by: ?ukasz Langa files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index a651d8829cd29..09211681baf30 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1346,7 +1346,10 @@ are always available. They are listed here in alphabetical order. operands, the result has the same type as the operands (after coercion) unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, ``pow(10, 2)`` - returns ``100``, but ``pow(10, -2)`` returns ``0.01``. + returns ``100``, but ``pow(10, -2)`` returns ``0.01``. For a negative base of + type :class:`int` or :class:`float` and a non-integral exponent, a complex + result is delivered. For example, ``pow(-9, 0.5)`` returns a value close + to ``3j``. For :class:`int` operands *base* and *exp*, if *mod* is present, *mod* must also be of integer type and *mod* must be nonzero. If *mod* is present and From webhook-mailer at python.org Thu Oct 21 16:59:27 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 20:59:27 -0000 Subject: [Python-checkins] bpo-45160: Ttk optionmenu only set variable once (GH-28291) (GH-29132) Message-ID: https://github.com/python/cpython/commit/04485ac9886cd807aae854fb905a88c791cfdde6 commit: 04485ac9886cd807aae854fb905a88c791cfdde6 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-21T22:59:20+02:00 summary: bpo-45160: Ttk optionmenu only set variable once (GH-28291) (GH-29132) (cherry picked from commit add46f84769a7e6fafa50954f79b7c248231fa4e) Co-authored-by: E-Paine <63801254+E-Paine at users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/ttk.py diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index 438d21d0b3733..cddd1f2e84834 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -301,6 +301,19 @@ def test_unique_radiobuttons(self): optmenu.destroy() optmenu2.destroy() + def test_trace_variable(self): + # prior to bpo45160, tracing a variable would cause the callback to be made twice + success = [] + items = ('a', 'b', 'c') + textvar = tkinter.StringVar(self.root) + def cb_test(*args): + self.assertEqual(textvar.get(), items[1]) + success.append(True) + optmenu = ttk.OptionMenu(self.root, textvar, "a", *items) + textvar.trace("w", cb_test) + optmenu['menu'].invoke(1) + self.assertEqual(success, [True]) + class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index ab7aeb15e8ff2..9b58497251d38 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -1643,7 +1643,10 @@ def set_menu(self, default=None, *values): menu.delete(0, 'end') for val in values: menu.add_radiobutton(label=val, - command=tkinter._setit(self._variable, val, self._callback), + command=( + None if self._callback is None + else lambda val=val: self._callback(val) + ), variable=self._variable) if default: diff --git a/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst new file mode 100644 index 0000000000000..9d11ed0e55d24 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst @@ -0,0 +1 @@ +When tracing a tkinter variable used by a ttk OptionMenu, callbacks are no longer made twice. \ No newline at end of file From webhook-mailer at python.org Thu Oct 21 17:02:28 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 21:02:28 -0000 Subject: [Python-checkins] bpo-45160: Ttk optionmenu only set variable once (GH-28291) (GH-29131) Message-ID: https://github.com/python/cpython/commit/98f157de1260801fd26c72eb6bfae0d9295e5ab3 commit: 98f157de1260801fd26c72eb6bfae0d9295e5ab3 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-21T23:02:20+02:00 summary: bpo-45160: Ttk optionmenu only set variable once (GH-28291) (GH-29131) (cherry picked from commit add46f84769a7e6fafa50954f79b7c248231fa4e) Co-authored-by: E-Paine <63801254+E-Paine at users.noreply.github.com> files: A Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst M Lib/tkinter/test/test_ttk/test_extensions.py M Lib/tkinter/ttk.py diff --git a/Lib/tkinter/test/test_ttk/test_extensions.py b/Lib/tkinter/test/test_ttk/test_extensions.py index 438d21d0b3733..cddd1f2e84834 100644 --- a/Lib/tkinter/test/test_ttk/test_extensions.py +++ b/Lib/tkinter/test/test_ttk/test_extensions.py @@ -301,6 +301,19 @@ def test_unique_radiobuttons(self): optmenu.destroy() optmenu2.destroy() + def test_trace_variable(self): + # prior to bpo45160, tracing a variable would cause the callback to be made twice + success = [] + items = ('a', 'b', 'c') + textvar = tkinter.StringVar(self.root) + def cb_test(*args): + self.assertEqual(textvar.get(), items[1]) + success.append(True) + optmenu = ttk.OptionMenu(self.root, textvar, "a", *items) + textvar.trace("w", cb_test) + optmenu['menu'].invoke(1) + self.assertEqual(success, [True]) + class DefaultRootTest(AbstractDefaultRootTest, unittest.TestCase): diff --git a/Lib/tkinter/ttk.py b/Lib/tkinter/ttk.py index b854235a62679..acdd565ec48a9 100644 --- a/Lib/tkinter/ttk.py +++ b/Lib/tkinter/ttk.py @@ -1643,7 +1643,10 @@ def set_menu(self, default=None, *values): menu.delete(0, 'end') for val in values: menu.add_radiobutton(label=val, - command=tkinter._setit(self._variable, val, self._callback), + command=( + None if self._callback is None + else lambda val=val: self._callback(val) + ), variable=self._variable) if default: diff --git a/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst new file mode 100644 index 0000000000000..9d11ed0e55d24 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-11-14-47-05.bpo-45160.VzMXbW.rst @@ -0,0 +1 @@ +When tracing a tkinter variable used by a ttk OptionMenu, callbacks are no longer made twice. \ No newline at end of file From webhook-mailer at python.org Thu Oct 21 17:05:55 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 21:05:55 -0000 Subject: [Python-checkins] bpo-45526: obmalloc radix use 64 addr bits (GH-29062) Message-ID: https://github.com/python/cpython/commit/0224b7180b280794b9fba62057b278ffb536c86f commit: 0224b7180b280794b9fba62057b278ffb536c86f branch: main author: Neil Schemenauer committer: ambv date: 2021-10-21T23:05:46+02:00 summary: bpo-45526: obmalloc radix use 64 addr bits (GH-29062) Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst M Objects/obmalloc.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst new file mode 100644 index 0000000000000..c35e5f58a4113 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-10-29-47.bpo-45526.WQnvW9.rst @@ -0,0 +1,3 @@ +In obmalloc, set ADDRESS_BITS to not ignore any bits (ignored 16 before). That is +safer in the case that the kernel gives user-space virtual addresses that span +a range greater than 48 bits. diff --git a/Objects/obmalloc.c b/Objects/obmalloc.c index 2eddb2cafd498..4e17bf44b4e96 100644 --- a/Objects/obmalloc.c +++ b/Objects/obmalloc.c @@ -1280,21 +1280,30 @@ _Py_GetAllocatedBlocks(void) #if WITH_PYMALLOC_RADIX_TREE /*==========================================================================*/ -/* radix tree for tracking arena usage +/* radix tree for tracking arena usage. If enabled, used to implement + address_in_range(). - bit allocation for keys + memory address bit allocation for keys - 64-bit pointers and 2^20 arena size: - 16 -> ignored (POINTER_BITS - ADDRESS_BITS) - 10 -> MAP_TOP - 10 -> MAP_MID - 8 -> MAP_BOT + 64-bit pointers, IGNORE_BITS=0 and 2^20 arena size: + 15 -> MAP_TOP_BITS + 15 -> MAP_MID_BITS + 14 -> MAP_BOT_BITS + 20 -> ideal aligned arena + ---- + 64 + + 64-bit pointers, IGNORE_BITS=16, and 2^20 arena size: + 16 -> IGNORE_BITS + 10 -> MAP_TOP_BITS + 10 -> MAP_MID_BITS + 8 -> MAP_BOT_BITS 20 -> ideal aligned arena ---- 64 32-bit pointers and 2^18 arena size: - 14 -> MAP_BOT + 14 -> MAP_BOT_BITS 18 -> ideal aligned arena ---- 32 @@ -1306,11 +1315,16 @@ _Py_GetAllocatedBlocks(void) /* number of bits in a pointer */ #define POINTER_BITS 64 -/* Current 64-bit processors are limited to 48-bit physical addresses. For - * now, the top 17 bits of addresses will all be equal to bit 2**47. If that - * changes in the future, this must be adjusted upwards. +/* High bits of memory addresses that will be ignored when indexing into the + * radix tree. Setting this to zero is the safe default. For most 64-bit + * machines, setting this to 16 would be safe. The kernel would not give + * user-space virtual memory addresses that have significant information in + * those high bits. The main advantage to setting IGNORE_BITS > 0 is that less + * virtual memory will be used for the top and middle radix tree arrays. Those + * arrays are allocated in the BSS segment and so will typically consume real + * memory only if actually accessed. */ -#define ADDRESS_BITS 48 +#define IGNORE_BITS 0 /* use the top and mid layers of the radix tree */ #define USE_INTERIOR_NODES @@ -1318,7 +1332,7 @@ _Py_GetAllocatedBlocks(void) #elif SIZEOF_VOID_P == 4 #define POINTER_BITS 32 -#define ADDRESS_BITS 32 +#define IGNORE_BITS 0 #else @@ -1332,6 +1346,9 @@ _Py_GetAllocatedBlocks(void) # error "arena size must be < 2^32" #endif +/* the lower bits of the address that are not ignored */ +#define ADDRESS_BITS (POINTER_BITS - IGNORE_BITS) + #ifdef USE_INTERIOR_NODES /* number of bits used for MAP_TOP and MAP_MID nodes */ #define INTERIOR_BITS ((ADDRESS_BITS - ARENA_BITS + 2) / 3) @@ -1360,11 +1377,9 @@ _Py_GetAllocatedBlocks(void) #define MAP_MID_INDEX(p) ((AS_UINT(p) >> MAP_MID_SHIFT) & MAP_MID_MASK) #define MAP_TOP_INDEX(p) ((AS_UINT(p) >> MAP_TOP_SHIFT) & MAP_TOP_MASK) -#if ADDRESS_BITS > POINTER_BITS -/* Return non-physical address bits of a pointer. Those bits should be same - * for all valid pointers if ADDRESS_BITS set correctly. Linux has support for - * 57-bit address space (Intel 5-level paging) but will not currently give - * those addresses to user space. +#if IGNORE_BITS > 0 +/* Return the ignored part of the pointer address. Those bits should be same + * for all valid pointers if IGNORE_BITS is set correctly. */ #define HIGH_BITS(p) (AS_UINT(p) >> ADDRESS_BITS) #else @@ -1416,7 +1431,7 @@ static arena_map_bot_t * arena_map_get(block *p, int create) { #ifdef USE_INTERIOR_NODES - /* sanity check that ADDRESS_BITS is correct */ + /* sanity check that IGNORE_BITS is correct */ assert(HIGH_BITS(p) == HIGH_BITS(&arena_map_root)); int i1 = MAP_TOP_INDEX(p); if (arena_map_root.ptrs[i1] == NULL) { @@ -1476,7 +1491,7 @@ arena_map_get(block *p, int create) static int arena_map_mark_used(uintptr_t arena_base, int is_used) { - /* sanity check that ADDRESS_BITS is correct */ + /* sanity check that IGNORE_BITS is correct */ assert(HIGH_BITS(arena_base) == HIGH_BITS(&arena_map_root)); arena_map_bot_t *n_hi = arena_map_get((block *)arena_base, is_used); if (n_hi == NULL) { From webhook-mailer at python.org Thu Oct 21 17:13:46 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 21:13:46 -0000 Subject: [Python-checkins] bpo-41983: add availability info to socket docs (GH-27519) Message-ID: https://github.com/python/cpython/commit/51375388bee7287be2d942906b48c8cf3f691e8b commit: 51375388bee7287be2d942906b48c8cf3f691e8b branch: main author: andrei kulakov committer: ambv date: 2021-10-21T23:13:37+02:00 summary: bpo-41983: add availability info to socket docs (GH-27519) * add availability info to AF_PACKET section * add availability for AF_QIPCRTR as well files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index 27ad5c7fd1d91..831821bbda77c 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -197,11 +197,15 @@ created. Socket addresses are represented as follows: - *addr* - Optional bytes-like object specifying the hardware physical address, whose interpretation depends on the device. + .. availability:: Linux >= 2.2. + - :const:`AF_QIPCRTR` is a Linux-only socket based interface for communicating with services running on co-processors in Qualcomm platforms. The address family is represented as a ``(node, port)`` tuple where the *node* and *port* are non-negative integers. + .. availability:: Linux >= 4.7. + .. versionadded:: 3.8 - :const:`IPPROTO_UDPLITE` is a variant of UDP which allows you to specify From webhook-mailer at python.org Thu Oct 21 17:17:07 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 21 Oct 2021 21:17:07 -0000 Subject: [Python-checkins] bpo-45557: Fix underscore_numbers in pprint.pprint(). (GH-29129) Message-ID: https://github.com/python/cpython/commit/6b75ad5fd47e5b34a04197927f748d0391898de7 commit: 6b75ad5fd47e5b34a04197927f748d0391898de7 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-21T14:16:59-07:00 summary: bpo-45557: Fix underscore_numbers in pprint.pprint(). (GH-29129) (cherry picked from commit 087f089e5e04d5b132ffbff0576667d591f13219) Co-authored-by: Eric V. Smith files: A Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst M Lib/pprint.py diff --git a/Lib/pprint.py b/Lib/pprint.py index 13819f3fef212..d91421f0a6bf6 100644 --- a/Lib/pprint.py +++ b/Lib/pprint.py @@ -50,7 +50,8 @@ def pprint(object, stream=None, indent=1, width=80, depth=None, *, """Pretty-print a Python object to a stream [default is sys.stdout].""" printer = PrettyPrinter( stream=stream, indent=indent, width=width, depth=depth, - compact=compact, sort_dicts=sort_dicts, underscore_numbers=False) + compact=compact, sort_dicts=sort_dicts, + underscore_numbers=underscore_numbers) printer.pprint(object) def pformat(object, indent=1, width=80, depth=None, *, diff --git a/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst b/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst new file mode 100644 index 0000000000000..7472b08d1b08e --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-21-16-18-51.bpo-45557.4MQt4r.rst @@ -0,0 +1,2 @@ +pprint.pprint() now handles underscore_numbers correctly. Previously it was +always setting it to False. From webhook-mailer at python.org Thu Oct 21 18:07:16 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 22:07:16 -0000 Subject: [Python-checkins] bpo-41983: add availability info to socket docs (GH-27519) (GH-29136) Message-ID: https://github.com/python/cpython/commit/8fd7e8965d0ff9f76401b86aec2166cc59a40637 commit: 8fd7e8965d0ff9f76401b86aec2166cc59a40637 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-22T00:07:07+02:00 summary: bpo-41983: add availability info to socket docs (GH-27519) (GH-29136) * add availability info to AF_PACKET section * add availability for AF_QIPCRTR as well (cherry picked from commit 51375388bee7287be2d942906b48c8cf3f691e8b) Co-authored-by: andrei kulakov files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index b16bf22348fdf4..c69a50266f60fa 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -197,11 +197,15 @@ created. Socket addresses are represented as follows: - *addr* - Optional bytes-like object specifying the hardware physical address, whose interpretation depends on the device. + .. availability:: Linux >= 2.2. + - :const:`AF_QIPCRTR` is a Linux-only socket based interface for communicating with services running on co-processors in Qualcomm platforms. The address family is represented as a ``(node, port)`` tuple where the *node* and *port* are non-negative integers. + .. availability:: Linux >= 4.7. + .. versionadded:: 3.8 - :const:`IPPROTO_UDPLITE` is a variant of UDP which allows you to specify From webhook-mailer at python.org Thu Oct 21 18:07:43 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 22:07:43 -0000 Subject: [Python-checkins] bpo-41983: add availability info to socket docs (GH-27519) (GH-29137) Message-ID: https://github.com/python/cpython/commit/b26eae54e98f44eebc1599ca458228dae0568ffa commit: b26eae54e98f44eebc1599ca458228dae0568ffa branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-22T00:07:38+02:00 summary: bpo-41983: add availability info to socket docs (GH-27519) (GH-29137) * add availability info to AF_PACKET section * add availability for AF_QIPCRTR as well (cherry picked from commit 51375388bee7287be2d942906b48c8cf3f691e8b) Co-authored-by: andrei kulakov files: M Doc/library/socket.rst diff --git a/Doc/library/socket.rst b/Doc/library/socket.rst index bc8723a90daad..db94a2a124575 100755 --- a/Doc/library/socket.rst +++ b/Doc/library/socket.rst @@ -197,11 +197,15 @@ created. Socket addresses are represented as follows: - *addr* - Optional bytes-like object specifying the hardware physical address, whose interpretation depends on the device. + .. availability:: Linux >= 2.2. + - :const:`AF_QIPCRTR` is a Linux-only socket based interface for communicating with services running on co-processors in Qualcomm platforms. The address family is represented as a ``(node, port)`` tuple where the *node* and *port* are non-negative integers. + .. availability:: Linux >= 4.7. + .. versionadded:: 3.8 - :const:`IPPROTO_UDPLITE` is a variant of UDP which allows you to specify From webhook-mailer at python.org Thu Oct 21 18:08:26 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 22:08:26 -0000 Subject: [Python-checkins] bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) (GH-29135) Message-ID: https://github.com/python/cpython/commit/9b3cda56870d087cf50f605e91f3d26964868640 commit: 9b3cda56870d087cf50f605e91f3d26964868640 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-22T00:08:20+02:00 summary: bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) (GH-29135) Co-authored-by: ?ukasz Langa (cherry picked from commit 887a55705bb6c05a507c2886c9978a9e0cff0dd7) Co-authored-by: Mark Dickinson files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index f4c3ef4600f6b..689455409eec3 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1357,7 +1357,10 @@ are always available. They are listed here in alphabetical order. operands, the result has the same type as the operands (after coercion) unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, ``pow(10, 2)`` - returns ``100``, but ``pow(10, -2)`` returns ``0.01``. + returns ``100``, but ``pow(10, -2)`` returns ``0.01``. For a negative base of + type :class:`int` or :class:`float` and a non-integral exponent, a complex + result is delivered. For example, ``pow(-9, 0.5)`` returns a value close + to ``3j``. For :class:`int` operands *base* and *exp*, if *mod* is present, *mod* must also be of integer type and *mod* must be nonzero. If *mod* is present and From webhook-mailer at python.org Thu Oct 21 18:08:40 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 22:08:40 -0000 Subject: [Python-checkins] bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) (GH-29134) Message-ID: https://github.com/python/cpython/commit/c53428fe8980aab6eda3e573bafed657e6798e6e commit: c53428fe8980aab6eda3e573bafed657e6798e6e branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-22T00:08:36+02:00 summary: bpo-44344: Document that pow can return a complex number for non-complex inputs. (GH-27853) (GH-29134) Co-authored-by: ?ukasz Langa (cherry picked from commit 887a55705bb6c05a507c2886c9978a9e0cff0dd7) Co-authored-by: Mark Dickinson files: M Doc/library/functions.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index b17ca69760dbc..942ad6fab85c9 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -1299,7 +1299,10 @@ are always available. They are listed here in alphabetical order. operands, the result has the same type as the operands (after coercion) unless the second argument is negative; in that case, all arguments are converted to float and a float result is delivered. For example, ``pow(10, 2)`` - returns ``100``, but ``pow(10, -2)`` returns ``0.01``. + returns ``100``, but ``pow(10, -2)`` returns ``0.01``. For a negative base of + type :class:`int` or :class:`float` and a non-integral exponent, a complex + result is delivered. For example, ``pow(-9, 0.5)`` returns a value close + to ``3j``. For :class:`int` operands *base* and *exp*, if *mod* is present, *mod* must also be of integer type and *mod* must be nonzero. If *mod* is present and From webhook-mailer at python.org Thu Oct 21 18:09:52 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 21 Oct 2021 22:09:52 -0000 Subject: [Python-checkins] bpo-44547: Make Fractions objects instances of typing.SupportsInt (GH-27851) Message-ID: https://github.com/python/cpython/commit/d1b24775b462f4f28aa4929fd031899170793388 commit: d1b24775b462f4f28aa4929fd031899170793388 branch: main author: Mark Dickinson committer: ambv date: 2021-10-22T00:09:47+02:00 summary: bpo-44547: Make Fractions objects instances of typing.SupportsInt (GH-27851) Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-08-20-10-52-40.bpo-44547.eu0iJq.rst M Doc/library/fractions.rst M Doc/whatsnew/3.11.rst M Lib/fractions.py M Lib/test/test_fractions.py diff --git a/Doc/library/fractions.rst b/Doc/library/fractions.rst index d04de8f8e95a6..c893f2d5389d5 100644 --- a/Doc/library/fractions.rst +++ b/Doc/library/fractions.rst @@ -94,6 +94,10 @@ another rational number, or from a string. Underscores are now permitted when creating a :class:`Fraction` instance from a string, following :PEP:`515` rules. + .. versionchanged:: 3.11 + :class:`Fraction` implements ``__int__`` now to satisfy + ``typing.SupportsInt`` instance checks. + .. attribute:: numerator Numerator of the Fraction in lowest term. diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 74fc7536ea231..a03fff8a9e10a 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -193,8 +193,12 @@ Improved Modules fractions --------- -Support :PEP:`515`-style initialization of :class:`~fractions.Fraction` from -string. (Contributed by Sergey B Kirpichev in :issue:`44258`.) +* Support :PEP:`515`-style initialization of :class:`~fractions.Fraction` from + string. (Contributed by Sergey B Kirpichev in :issue:`44258`.) + +* :class:`~fractions.Fraction` now implements an ``__int__`` method, so + that an ``isinstance(some_fraction, typing.SupportsInt)`` check passes. + (Contributed by Mark Dickinson in :issue:`44547`.) math diff --git a/Lib/fractions.py b/Lib/fractions.py index 180cd94c2879c..f9ac882ec002f 100644 --- a/Lib/fractions.py +++ b/Lib/fractions.py @@ -594,8 +594,15 @@ def __abs__(a): """abs(a)""" return Fraction(abs(a._numerator), a._denominator, _normalize=False) + def __int__(a, _index=operator.index): + """int(a)""" + if a._numerator < 0: + return _index(-(-a._numerator // a._denominator)) + else: + return _index(a._numerator // a._denominator) + def __trunc__(a): - """trunc(a)""" + """math.trunc(a)""" if a._numerator < 0: return -(-a._numerator // a._denominator) else: diff --git a/Lib/test/test_fractions.py b/Lib/test/test_fractions.py index bbf7709fe959b..fc46e8674fc46 100644 --- a/Lib/test/test_fractions.py +++ b/Lib/test/test_fractions.py @@ -8,6 +8,7 @@ import fractions import functools import sys +import typing import unittest from copy import copy, deepcopy import pickle @@ -385,6 +386,47 @@ def testConversions(self): self.assertTypedEquals(0.1+0j, complex(F(1,10))) + def testSupportsInt(self): + # See bpo-44547. + f = F(3, 2) + self.assertIsInstance(f, typing.SupportsInt) + self.assertEqual(int(f), 1) + self.assertEqual(type(int(f)), int) + + def testIntGuaranteesIntReturn(self): + # Check that int(some_fraction) gives a result of exact type `int` + # even if the fraction is using some other Integral type for its + # numerator and denominator. + + class CustomInt(int): + """ + Subclass of int with just enough machinery to convince the Fraction + constructor to produce something with CustomInt numerator and + denominator. + """ + + @property + def numerator(self): + return self + + @property + def denominator(self): + return CustomInt(1) + + def __mul__(self, other): + return CustomInt(int(self) * int(other)) + + def __floordiv__(self, other): + return CustomInt(int(self) // int(other)) + + f = F(CustomInt(13), CustomInt(5)) + + self.assertIsInstance(f.numerator, CustomInt) + self.assertIsInstance(f.denominator, CustomInt) + self.assertIsInstance(f, typing.SupportsInt) + self.assertEqual(int(f), 2) + self.assertEqual(type(int(f)), int) + def testBoolGuarateesBoolReturn(self): # Ensure that __bool__ is used on numerator which guarantees a bool # return. See also bpo-39274. diff --git a/Misc/NEWS.d/next/Library/2021-08-20-10-52-40.bpo-44547.eu0iJq.rst b/Misc/NEWS.d/next/Library/2021-08-20-10-52-40.bpo-44547.eu0iJq.rst new file mode 100644 index 0000000000000..a5f425e17cad2 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-08-20-10-52-40.bpo-44547.eu0iJq.rst @@ -0,0 +1,2 @@ +Implement ``Fraction.__int__``, so that a :class:`fractions.Fraction` +instance ``f`` passes an ``isinstance(f, typing.SupportsInt)`` check. From webhook-mailer at python.org Thu Oct 21 18:58:09 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 21 Oct 2021 22:58:09 -0000 Subject: [Python-checkins] bpo-44019: Add test_all_exported_names for operator module (GH-29124) Message-ID: https://github.com/python/cpython/commit/37fad7d3b7154c44b9902a2ab0db8641f1a0284b commit: 37fad7d3b7154c44b9902a2ab0db8641f1a0284b branch: main author: Dong-hee Na committer: corona10 date: 2021-10-22T07:58:04+09:00 summary: bpo-44019: Add test_all_exported_names for operator module (GH-29124) files: M Lib/test/test_operator.py diff --git a/Lib/test/test_operator.py b/Lib/test/test_operator.py index cf3439fe6fb82..b7e38c2334987 100644 --- a/Lib/test/test_operator.py +++ b/Lib/test/test_operator.py @@ -45,6 +45,18 @@ def __iter__(self): class OperatorTestCase: + def test___all__(self): + operator = self.module + actual_all = set(operator.__all__) + computed_all = set() + for name in vars(operator): + if name.startswith('__'): + continue + value = getattr(operator, name) + if value.__module__ in ('operator', '_operator'): + computed_all.add(name) + self.assertSetEqual(computed_all, actual_all) + def test_lt(self): operator = self.module self.assertRaises(TypeError, operator.lt) From webhook-mailer at python.org Thu Oct 21 19:20:51 2021 From: webhook-mailer at python.org (corona10) Date: Thu, 21 Oct 2021 23:20:51 -0000 Subject: [Python-checkins] bpo-43706: Use PEP 590 vectorcall to speed up enumerate() (GH-25154) Message-ID: https://github.com/python/cpython/commit/83f202a802b0810e2f0900c6d298fd104390f2ba commit: 83f202a802b0810e2f0900c6d298fd104390f2ba branch: main author: Dong-hee Na committer: corona10 date: 2021-10-22T08:20:43+09:00 summary: bpo-43706: Use PEP 590 vectorcall to speed up enumerate() (GH-25154) files: A Misc/NEWS.d/next/Core and Builtins/2021-04-03-02-44-15.bpo-43706.jjsXlT.rst M Objects/enumobject.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-04-03-02-44-15.bpo-43706.jjsXlT.rst b/Misc/NEWS.d/next/Core and Builtins/2021-04-03-02-44-15.bpo-43706.jjsXlT.rst new file mode 100644 index 0000000000000..ee859e37911cd --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-04-03-02-44-15.bpo-43706.jjsXlT.rst @@ -0,0 +1,2 @@ +Speed up calls to ``enumerate()`` by using the :pep:`590` ``vectorcall`` +calling convention. Patch by Dong-hee Na. diff --git a/Objects/enumobject.c b/Objects/enumobject.c index 4513831545b9a..b78230ddaebaa 100644 --- a/Objects/enumobject.c +++ b/Objects/enumobject.c @@ -81,6 +81,45 @@ enum_new_impl(PyTypeObject *type, PyObject *iterable, PyObject *start) return (PyObject *)en; } +// TODO: Use AC when bpo-43447 is supported +static PyObject * +enumerate_vectorcall(PyObject *type, PyObject *const *args, + size_t nargsf, PyObject *kwnames) +{ + assert(PyType_Check(type)); + PyTypeObject *tp = (PyTypeObject *)type; + Py_ssize_t nargs = PyVectorcall_NARGS(nargsf); + Py_ssize_t nkwargs = 0; + if (nargs == 0) { + PyErr_SetString(PyExc_TypeError, + "enumerate() missing required argument 'iterable'"); + return NULL; + } + if (kwnames != NULL) { + nkwargs = PyTuple_GET_SIZE(kwnames); + } + + if (nargs + nkwargs == 2) { + if (nkwargs == 1) { + PyObject *kw = PyTuple_GET_ITEM(kwnames, 0); + if (!_PyUnicode_EqualToASCIIString(kw, "start")) { + PyErr_Format(PyExc_TypeError, + "'%S' is an invalid keyword argument for enumerate()", kw); + return NULL; + } + } + return enum_new_impl(tp, args[0], args[1]); + } + + if (nargs == 1 && nkwargs == 0) { + return enum_new_impl(tp, args[0], NULL); + } + + PyErr_Format(PyExc_TypeError, + "enumerate() takes at most 2 arguments (%d given)", nargs + nkwargs); + return NULL; +} + static void enum_dealloc(enumobject *en) { @@ -261,6 +300,7 @@ PyTypeObject PyEnum_Type = { PyType_GenericAlloc, /* tp_alloc */ enum_new, /* tp_new */ PyObject_GC_Del, /* tp_free */ + .tp_vectorcall = (vectorcallfunc)enumerate_vectorcall }; /* Reversed Object ***************************************************************/ From webhook-mailer at python.org Thu Oct 21 20:02:20 2021 From: webhook-mailer at python.org (zware) Date: Fri, 22 Oct 2021 00:02:20 -0000 Subject: [Python-checkins] bpo-29844: Remove obsolete paragraph from Tools/msi/README.txt (GH-29141) Message-ID: https://github.com/python/cpython/commit/098a33f6a60ca2df88e03024ac9d8da507cfa5c8 commit: 098a33f6a60ca2df88e03024ac9d8da507cfa5c8 branch: main author: Zachary Ware committer: zware date: 2021-10-21T19:02:12-05:00 summary: bpo-29844: Remove obsolete paragraph from Tools/msi/README.txt (GH-29141) files: M Tools/msi/README.txt diff --git a/Tools/msi/README.txt b/Tools/msi/README.txt index c85ee4564d838..0a4382f3e0223 100644 --- a/Tools/msi/README.txt +++ b/Tools/msi/README.txt @@ -363,6 +363,8 @@ of Python's files. Within this install directory is the following approximate layout: .\python[w].exe The core executable files +.\python3x.dll The core interpreter +.\python3.dll The stable ABI reference .\DLLs Stdlib extensions (*.pyd) and dependencies .\Doc Documentation (*.chm) .\include Development headers (*.h) @@ -373,14 +375,6 @@ Within this install directory is the following approximate layout: .\tcl Tcl dependencies (*.dll, *.tcl and others) .\Tools Tool scripts (*.py) -When installed for all users, the following files are installed to -either "%SystemRoot%\System32" or "%SystemRoot%\SysWOW64" as -appropriate. For the current user, they are installed in the Python -install directory. - -.\python3x.dll The core interpreter -.\python3.dll The stable ABI reference - When installed for all users, the following files are installed to "%SystemRoot%" (typically "C:\Windows") to ensure they are always available on PATH. (See Launching Python below.) For the current user, @@ -388,6 +382,7 @@ they are installed in "%LocalAppData%\Programs\Python\PyLauncher". .\py[w].exe PEP 397 launcher + System Settings =============== From webhook-mailer at python.org Thu Oct 21 20:07:34 2021 From: webhook-mailer at python.org (brettcannon) Date: Fri, 22 Oct 2021 00:07:34 -0000 Subject: [Python-checkins] bpo-45548: Make `Modules/Setup` easier to read (GH-29143) Message-ID: https://github.com/python/cpython/commit/01cf4fb9c1aa567484c2ffb1b11f9b3fe9949b05 commit: 01cf4fb9c1aa567484c2ffb1b11f9b3fe9949b05 branch: main author: Brett Cannon committer: brettcannon date: 2021-10-21T17:07:26-07:00 summary: bpo-45548: Make `Modules/Setup` easier to read (GH-29143) files: M Modules/Setup diff --git a/Modules/Setup b/Modules/Setup index 2e44ec24dacdc..a4440dfb6fc9d 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -62,9 +62,9 @@ # *noconfig* has the same effect as *shared*.) # # NOTE: As a standard policy, as many modules as can be supported by a -# platform should be present. The distribution comes with all modules -# enabled that are supported by most platforms and don't require you -# to ftp sources from elsewhere. +# platform should be listed below. The distribution comes with all +# modules enabled that are supported by most platforms and don't +# require you to download sources from elsewhere. # Some special rules to define PYTHONPATH. @@ -94,58 +94,40 @@ COREPYTHONPATH=$(DESTPATH)$(SITEPATH)$(TESTPATH) PYTHONPATH=$(COREPYTHONPATH) -# The modules listed here can't be built as shared libraries for -# various reasons; therefore they are listed here instead of in the -# normal order. - -# This only contains the minimal set of modules required to run the -# setup.py script in the root of the Python source tree. - -posix -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal posixmodule.c # posix (UNIX) system calls -errno errnomodule.c # posix (UNIX) errno values -pwd pwdmodule.c # this is needed to find out the user's home dir - # if $HOME is not set -_sre -DPy_BUILD_CORE_BUILTIN _sre.c # Fredrik Lundh's new regular expressions -_codecs _codecsmodule.c # access to the builtin codecs and codec registry -_weakref _weakref.c # weak references -_functools -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _functoolsmodule.c # Tools for working with functions and callable objects -_operator -DPy_BUILD_CORE_BUILTIN _operator.c # operator.add() and similar goodies -_collections _collectionsmodule.c # Container types -_abc -DPy_BUILD_CORE_BUILTIN _abc.c # Abstract base classes -itertools itertoolsmodule.c # Functions creating iterators for efficient looping -atexit atexitmodule.c # Register functions to be run at interpreter-shutdown -_signal -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal signalmodule.c -_stat _stat.c # stat.h interface -time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c # -lm # time operations and variables -_thread -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _threadmodule.c # low-level threading interface - -# access to ISO C locale support -_locale -DPy_BUILD_CORE_BUILTIN _localemodule.c # -lintl +# --- +# Built-in modules required to get a functioning interpreter; +# cannot be built as shared! -# Standard I/O baseline +_collections _collectionsmodule.c +_abc -DPy_BUILD_CORE_BUILTIN _abc.c +_codecs _codecsmodule.c +_functools -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _functoolsmodule.c _io -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c - -# faulthandler module +_locale -DPy_BUILD_CORE_BUILTIN _localemodule.c # -lintl +_operator -DPy_BUILD_CORE_BUILTIN _operator.c +_signal -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal signalmodule.c +_sre -DPy_BUILD_CORE_BUILTIN _sre.c +_stat _stat.c +_symtable symtablemodule.c # setup.py can't track the .h file that _symtable depends on. +_thread -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _threadmodule.c +_tracemalloc _tracemalloc.c # See bpo-35053 as to why this is built in. +_weakref _weakref.c +atexit atexitmodule.c +errno errnomodule.c faulthandler faulthandler.c +itertools itertoolsmodule.c +posix -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal posixmodule.c +pwd pwdmodule.c +time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c -# debug tool to trace memory blocks allocated by Python -# -# bpo-35053: The module must be builtin since _Py_NewReference() -# can call _PyTraceMalloc_NewReference(). -_tracemalloc _tracemalloc.c +# --- # The rest of the modules listed in this file are all commented out by # default. Usually they can be detected and built as dynamically -# loaded modules by the new setup.py script added in Python 2.1. If -# you're on a platform that doesn't support dynamic loading, want to -# compile modules statically into the Python binary, or need to -# specify some odd set of compiler switches, you can uncomment the -# appropriate lines below. - -# ====================================================================== - -# The Python symtable module depends on .h files that setup.py doesn't track -_symtable symtablemodule.c +# loaded modules by setup.py. If you're on a platform that doesn't +# support dynamic loading, want to compile modules statically into the +# Python binary, or need to specify some odd set of compiler switches, +# you can uncomment the appropriate lines below. # Uncommenting the following line tells makesetup that all following # modules are to be built as shared libraries (see above for more @@ -153,67 +135,91 @@ _symtable symtablemodule.c #*shared* -# GNU readline. Unlike previous Python incarnations, GNU readline is -# now incorporated in an optional module, configured in the Setup file -# instead of by a configure script switch. You may have to insert a -# -L option pointing to the directory where libreadline.* lives, -# and you may have to change -ltermcap to -ltermlib or perhaps remove -# it, depending on your system -- see the GNU readline instructions. -# It's okay for this to be a shared library, too. - -#readline readline.c -lreadline -ltermcap - - # Modules that should always be present (non UNIX dependent): -#array -DPy_BUILD_CORE_MODULE arraymodule.c # array objects -#cmath cmathmodule.c _math.c -DPy_BUILD_CORE_MODULE # -lm # complex math library functions -#math mathmodule.c _math.c -DPy_BUILD_CORE_MODULE # -lm # math library functions, e.g. sin() -#_contextvars _contextvarsmodule.c # Context Variables -#_struct -DPy_BUILD_CORE_MODULE _struct.c # binary structure packing/unpacking -#_testcapi _testcapimodule.c # Python C API test module; CANNOT be statically compiled! -#_testinternalcapi _testinternalcapi.c -I$(srcdir)/Include/internal -DPy_BUILD_CORE_MODULE # Python internal C API test module -#_random _randommodule.c -DPy_BUILD_CORE_MODULE # Random number generator -#_elementtree -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI _elementtree.c # elementtree accelerator -#_pickle -DPy_BUILD_CORE_MODULE _pickle.c # pickle accelerator -#_datetime _datetimemodule.c # datetime accelerator -#_zoneinfo _zoneinfo.c -DPy_BUILD_CORE_MODULE # zoneinfo accelerator -#_bisect _bisectmodule.c # Bisection algorithms -#_heapq _heapqmodule.c -DPy_BUILD_CORE_MODULE # Heap queue algorithm -#_asyncio _asynciomodule.c # Fast asyncio Future -#_json -I$(srcdir)/Include/internal -DPy_BUILD_CORE_BUILTIN _json.c # _json speedups -#_statistics _statisticsmodule.c # statistics accelerator -#_typing _typingmodule.c # typing accelerator -#_lsprof _lsprof.c rotatingtree.c # cProfile accelerators +#_asyncio _asynciomodule.c +#_bisect _bisectmodule.c +#_blake2 -DPy_BUILD_CORE_BUILTIN _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c +#_codecs_cn cjkcodecs/_codecs_cn.c +#_codecs_hk cjkcodecs/_codecs_hk.c +#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c +#_codecs_jp cjkcodecs/_codecs_jp.c +#_codecs_kr cjkcodecs/_codecs_kr.c +#_codecs_tw cjkcodecs/_codecs_tw.c +#_contextvars _contextvarsmodule.c +#_csv _csv.c +#_datetime _datetimemodule.c +#_elementtree -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat _elementtree.c +#_heapq -DPy_BUILD_CORE_MODULE _heapqmodule.c +#_json -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _json.c +#_lsprof _lsprof.c rotatingtree.c +#_md5 -DPy_BUILD_CORE_BUILTIN md5module.c +#_multibytecodec cjkcodecs/multibytecodec.c #_opcode _opcode.c -#_queue _queuemodule.c -DPy_BUILD_CORE_MODULE - -#unicodedata unicodedata.c -DPy_BUILD_CORE_BUILTIN # static Unicode character database - +#_pickle -DPy_BUILD_CORE_MODULE _pickle.c +#_posixsubprocess -DPy_BUILD_CORE_BUILTIN _posixsubprocess.c +#_queue -DPy_BUILD_CORE_MODULE _queuemodule.c +#_random -DPy_BUILD_CORE_MODULE _randommodule.c +#_sha1 -DPy_BUILD_CORE_BUILTIN sha1module.c +#_sha256 -DPy_BUILD_CORE_BUILTIN sha256module.c +#_sha512 -DPy_BUILD_CORE_BUILTIN sha512module.c +#_sha3 -DPy_BUILD_CORE_BUILTIN _sha3/sha3module.c +#_statistics _statisticsmodule.c +#_struct -DPy_BUILD_CORE_MODULE _struct.c +#_typing _typingmodule.c +#_zoneinfo -DPy_BUILD_CORE_MODULE _zoneinfo.c +#array -DPy_BUILD_CORE_MODULE arraymodule.c +#audioop audioop.c +#binascii -DPy_BUILD_CORE_MODULE binascii.c +#cmath -DPy_BUILD_CORE_MODULE cmathmodule.c _math.c # -lm +#math -DPy_BUILD_CORE_MODULE mathmodule.c _math.c # -lm +#pyexpat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c +#unicodedata -DPy_BUILD_CORE_BUILTIN unicodedata.c # Modules with some UNIX dependencies -- on by default: # (If you have a really backward UNIX, select and socket may not be # supported...) -#fcntl fcntlmodule.c # fcntl(2) and ioctl(2) -#spwd spwdmodule.c # spwd(3) -#grp grpmodule.c # grp(3) -#select selectmodule.c # select(2); not on ancient System V +#_socket socketmodule.c # socket(2) +#fcntl fcntlmodule.c # fcntl(2) and ioctl(2) +#grp grpmodule.c # grp(3) +#mmap mmapmodule.c # Also works on win32. #ossaudiodev ossaudiodev.c +#select selectmodule.c # select(2); not on ancient System V +#spwd spwdmodule.c # spwd(3) +#syslog syslogmodule.c -# Memory-mapped files (also works on Win32). -#mmap mmapmodule.c +# Some more UNIX dependent modules -- off by default, since these +# are not supported by all UNIX systems: -# CSV file helper -#_csv _csv.c +#_crypt _cryptmodule.c # -lcrypt # crypt(3); breaks many builds. +#nis nismodule.c -lnsl # Sun yellow pages -- not everywhere +#termios termios.c # Steen Lumholt's termios module +#resource resource.c # Jeremy Hylton's rlimit interface + +# Modules that require external libraries. + +#_bz2 _bz2module.c -lbz2 +#_dbm _dbmmodule.c # -lndbm # dbm(3) +#_gdbm -I/usr/local/include -L/usr/local/lib -lgdbm _gdbmmodule.c +#_lzma _lzmamodule.c -llzma +#zlib -I$(prefix)/include -L$(exec_prefix)/lib -lz zlibmodule.c + +# GNU readline. Unlike previous Python incarnations, GNU readline is +# now incorporated in an optional module, configured in the Setup file +# instead of by a configure script switch. You may have to insert a +# -L option pointing to the directory where libreadline.* lives, +# and you may have to change -ltermcap to -ltermlib or perhaps remove +# it, depending on your system -- see the GNU readline instructions. +# It's okay for this to be a shared library, too. + +#readline readline.c -lreadline -ltermcap -# Socket module helper for socket(2) -#_socket socketmodule.c +# Set OpenSSL when not using the system copy found by ./configure. +#OPENSSL=/path/to/openssl/directory -# Socket module helper for SSL support; you must comment out the other -# socket line above, and edit the OPENSSL variable: -# OPENSSL=/path/to/openssl/directory -# _ssl _ssl.c \ +# To dynamically link OpenSSL: +#_ssl _ssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ # -lssl -lcrypto \ # -DPy_BUILD_CORE_BUILTIN @@ -223,7 +229,7 @@ _symtable symtablemodule.c # -DPy_BUILD_CORE_BUILTIN # To statically link OpenSSL: -# _ssl _ssl.c \ +#_ssl _ssl.c \ # -I$(OPENSSL)/include -L$(OPENSSL)/lib \ # -l:libssl.a -Wl,--exclude-libs,libssl.a \ # -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ @@ -233,52 +239,6 @@ _symtable symtablemodule.c # -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ # -DPy_BUILD_CORE_BUILTIN -# The crypt module is now disabled by default because it breaks builds -# on many systems (where -lcrypt is needed), e.g. Linux (I believe). - -#_crypt _cryptmodule.c # -lcrypt # crypt(3); needs -lcrypt on some systems - - -# Some more UNIX dependent modules -- off by default, since these -# are not supported by all UNIX systems: - -#nis nismodule.c -lnsl # Sun yellow pages -- not everywhere -#termios termios.c # Steen Lumholt's termios module -#resource resource.c # Jeremy Hylton's rlimit interface - -#_posixsubprocess -DPy_BUILD_CORE_BUILTIN _posixsubprocess.c # POSIX subprocess module helper - -# Multimedia modules -- off by default. -# These don't work for 64-bit platforms!!! -# #993173 says audioop works on 64-bit platforms, though. -# These represent audio samples or images as strings: - -#audioop audioop.c # Operations on audio samples - - -# Note that the _md5 and _sha modules are normally only built if the -# system does not have the OpenSSL libs containing an optimized version. - -# The _md5 module implements the RSA Data Security, Inc. MD5 -# Message-Digest Algorithm, described in RFC 1321. - -#_md5 md5module.c -DPy_BUILD_CORE_BUILTIN - - -# The _sha module implements the SHA checksum algorithms. -# (NIST's Secure Hash Algorithms.) -#_sha1 sha1module.c -DPy_BUILD_CORE_BUILTIN -#_sha256 sha256module.c -DPy_BUILD_CORE_BUILTIN -#_sha512 sha512module.c -DPy_BUILD_CORE_BUILTIN -#_sha3 _sha3/sha3module.c -DPy_BUILD_CORE_BUILTIN - -# _blake module -#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c -DPy_BUILD_CORE_BUILTIN - -# Compression -#_bz2 _bz2module.c -lbz2 -#_lzma _lzmamodule.c -llzma - # The _tkinter module. # # The command for _tkinter is long and site specific. Please @@ -322,75 +282,34 @@ _symtable symtablemodule.c # *** Always uncomment this; X11 libraries to link with: # -lX11 -# Lance Ellinghaus's syslog module -#syslog syslogmodule.c # syslog daemon interface - - # Curses support, requiring the System V version of curses, often # provided by the ncurses library. e.g. on Linux, link with -lncurses # instead of -lcurses). -#_curses _cursesmodule.c -lcurses -ltermcap -DPy_BUILD_CORE_MODULE -# Wrapper for the panel library that's part of ncurses and SYSV curses. -#_curses_panel _curses_panel.c -lpanel -lncurses - - -# Modules that provide persistent dictionary-like semantics. You will -# probably want to arrange for at least one of them to be available on -# your machine, though none are defined by default because of library -# dependencies. The Python module dbm/__init__.py provides an -# implementation independent wrapper for these; dbm/dumb.py provides -# similar functionality (but slower of course) implemented in Python. - -#_dbm _dbmmodule.c # dbm(3) may require -lndbm or similar - -# Anthony Baxter's gdbm module. GNU dbm(3) will require -lgdbm: - -#_gdbm _gdbmmodule.c -I/usr/local/include -L/usr/local/lib -lgdbm - - -# Helper module for various ascii-encoders -#binascii binascii.c -DPy_BUILD_CORE_MODULE - -# Andrew Kuchling's zlib module. -# This require zlib 1.1.3 (or later). -# See http://www.gzip.org/zlib/ -#zlib zlibmodule.c -I$(prefix)/include -L$(exec_prefix)/lib -lz - -# Interface to the Expat XML parser -# More information on Expat can be found at www.libexpat.org. -# -#pyexpat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -I$(srcdir)/Modules/expat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY -DUSE_PYEXPAT_CAPI - -# Hye-Shik Chang's CJKCodecs +#_curses -DPy_BUILD_CORE_MODULE -lcurses -ltermcap _cursesmodule.c -# multibytecodec is required for all the other CJK codec modules -#_multibytecodec cjkcodecs/multibytecodec.c - -#_codecs_cn cjkcodecs/_codecs_cn.c -#_codecs_hk cjkcodecs/_codecs_hk.c -#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c -#_codecs_jp cjkcodecs/_codecs_jp.c -#_codecs_kr cjkcodecs/_codecs_kr.c -#_codecs_tw cjkcodecs/_codecs_tw.c - -# Example -- included for reference only: -# xx xxmodule.c +# Wrapper for the panel library that's part of ncurses and SYSV curses. +#_curses_panel -lpanel -lncurses _curses_panel.c -# Another example -- the 'xxsubtype' module shows C-level subtyping in action -xxsubtype xxsubtype.c +# Examples -# Limited API examples +#xx xxmodule.c #xxlimited xxlimited.c #xxlimited_35 xxlimited_35.c +xxsubtype xxsubtype.c # Required for the test suite to pass! + +# Testing -# For testing #_xxsubinterpreters _xxsubinterpretersmodule.c #_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c #_testbuffer _testbuffer.c +#_testcapi _testcapimodule.c # CANNOT be statically compiled! #_testimportmultiple _testimportmultiple.c +#_testinternalcapi -DPy_BUILD_CORE_MODULE -I$(srcdir)/Include/internal _testinternalcapi.c #_testmultiphase _testmultiphase.c + +# --- # Uncommenting the following line tells makesetup that all following modules # are not built (see above for more detail). # From webhook-mailer at python.org Thu Oct 21 21:22:35 2021 From: webhook-mailer at python.org (gpshead) Date: Fri, 22 Oct 2021 01:22:35 -0000 Subject: [Python-checkins] bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Message-ID: https://github.com/python/cpython/commit/843b890334ca30cf6af27dffe29cecd06b49f7d9 commit: 843b890334ca30cf6af27dffe29cecd06b49f7d9 branch: main author: Gregory P. Smith committer: gpshead date: 2021-10-21T18:22:25-07:00 summary: bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Raise RLIMIT_NOFILE in test.libregrtest. On macOS the default is often too low for our testsuite to succeed. Co-authored by reviewer: Victor Stinner files: A Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst M Lib/test/libregrtest/setup.py diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index f9460ae7ed18f..4ffd15478f643 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -39,6 +39,7 @@ def setup_tests(ns): for signum in signals: faulthandler.register(signum, chain=True, file=stderr_fd) + _adjust_resource_limits() replace_stdout() support.record_original_stdout(sys.stdout) @@ -133,3 +134,26 @@ def restore_stdout(): sys.stdout.close() sys.stdout = stdout atexit.register(restore_stdout) + + +def _adjust_resource_limits(): + """Adjust the system resource limits (ulimit) if needed.""" + try: + import resource + from resource import RLIMIT_NOFILE, RLIM_INFINITY + except ImportError: + return + fd_limit, max_fds = resource.getrlimit(RLIMIT_NOFILE) + # On macOS the default fd limit is sometimes too low (256) for our + # test suite to succeed. Raise it to something more reasonable. + # 1024 is a common Linux default. + desired_fds = 1024 + if fd_limit < desired_fds and fd_limit < max_fds: + new_fd_limit = min(desired_fds, max_fds) + try: + resource.setrlimit(RLIMIT_NOFILE, (new_fd_limit, max_fds)) + print(f"Raised RLIMIT_NOFILE: {fd_limit} -> {new_fd_limit}") + except (ValueError, OSError) as err: + print(f"Unable to raise RLIMIT_NOFILE from {fd_limit} to " + f"{new_fd_limit}: {err}.") + diff --git a/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst new file mode 100644 index 0000000000000..2528857caf9c0 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst @@ -0,0 +1,3 @@ +:mod:`test.libregrtest` now raises the soft resource limit for the maximum +number of file descriptors when the default is too low for our test suite as +was often the case on macOS. From webhook-mailer at python.org Fri Oct 22 00:47:19 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 04:47:19 -0000 Subject: [Python-checkins] bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Message-ID: https://github.com/python/cpython/commit/8f6aa48cb2dc827a2cb76e35e91bf02d099875c5 commit: 8f6aa48cb2dc827a2cb76e35e91bf02d099875c5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-21T21:47:07-07:00 summary: bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Raise RLIMIT_NOFILE in test.libregrtest. On macOS the default is often too low for our testsuite to succeed. Co-authored by reviewer: Victor Stinner (cherry picked from commit 843b890334ca30cf6af27dffe29cecd06b49f7d9) Co-authored-by: Gregory P. Smith files: A Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst M Lib/test/libregrtest/setup.py diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index 472c6ebec00cb..b79175944d2bd 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -39,6 +39,7 @@ def setup_tests(ns): for signum in signals: faulthandler.register(signum, chain=True, file=stderr_fd) + _adjust_resource_limits() replace_stdout() support.record_original_stdout(sys.stdout) @@ -134,3 +135,26 @@ def restore_stdout(): sys.stdout.close() sys.stdout = stdout atexit.register(restore_stdout) + + +def _adjust_resource_limits(): + """Adjust the system resource limits (ulimit) if needed.""" + try: + import resource + from resource import RLIMIT_NOFILE, RLIM_INFINITY + except ImportError: + return + fd_limit, max_fds = resource.getrlimit(RLIMIT_NOFILE) + # On macOS the default fd limit is sometimes too low (256) for our + # test suite to succeed. Raise it to something more reasonable. + # 1024 is a common Linux default. + desired_fds = 1024 + if fd_limit < desired_fds and fd_limit < max_fds: + new_fd_limit = min(desired_fds, max_fds) + try: + resource.setrlimit(RLIMIT_NOFILE, (new_fd_limit, max_fds)) + print(f"Raised RLIMIT_NOFILE: {fd_limit} -> {new_fd_limit}") + except (ValueError, OSError) as err: + print(f"Unable to raise RLIMIT_NOFILE from {fd_limit} to " + f"{new_fd_limit}: {err}.") + diff --git a/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst new file mode 100644 index 0000000000000..2528857caf9c0 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst @@ -0,0 +1,3 @@ +:mod:`test.libregrtest` now raises the soft resource limit for the maximum +number of file descriptors when the default is too low for our test suite as +was often the case on macOS. From webhook-mailer at python.org Fri Oct 22 00:48:48 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 04:48:48 -0000 Subject: [Python-checkins] bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Message-ID: https://github.com/python/cpython/commit/216c040bb1fdf73e78d67ab82a43563d7593f874 commit: 216c040bb1fdf73e78d67ab82a43563d7593f874 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-21T21:48:44-07:00 summary: bpo-43592: Raise RLIMIT_NOFILE in test.libregrtest (GH-29127) Raise RLIMIT_NOFILE in test.libregrtest. On macOS the default is often too low for our testsuite to succeed. Co-authored by reviewer: Victor Stinner (cherry picked from commit 843b890334ca30cf6af27dffe29cecd06b49f7d9) Co-authored-by: Gregory P. Smith files: A Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst M Lib/test/libregrtest/setup.py diff --git a/Lib/test/libregrtest/setup.py b/Lib/test/libregrtest/setup.py index 1f264c1be49fe..cedf905c35c55 100644 --- a/Lib/test/libregrtest/setup.py +++ b/Lib/test/libregrtest/setup.py @@ -35,6 +35,7 @@ def setup_tests(ns): for signum in signals: faulthandler.register(signum, chain=True, file=stderr_fd) + _adjust_resource_limits() replace_stdout() support.record_original_stdout(sys.stdout) @@ -117,3 +118,26 @@ def restore_stdout(): sys.stdout.close() sys.stdout = stdout atexit.register(restore_stdout) + + +def _adjust_resource_limits(): + """Adjust the system resource limits (ulimit) if needed.""" + try: + import resource + from resource import RLIMIT_NOFILE, RLIM_INFINITY + except ImportError: + return + fd_limit, max_fds = resource.getrlimit(RLIMIT_NOFILE) + # On macOS the default fd limit is sometimes too low (256) for our + # test suite to succeed. Raise it to something more reasonable. + # 1024 is a common Linux default. + desired_fds = 1024 + if fd_limit < desired_fds and fd_limit < max_fds: + new_fd_limit = min(desired_fds, max_fds) + try: + resource.setrlimit(RLIMIT_NOFILE, (new_fd_limit, max_fds)) + print(f"Raised RLIMIT_NOFILE: {fd_limit} -> {new_fd_limit}") + except (ValueError, OSError) as err: + print(f"Unable to raise RLIMIT_NOFILE from {fd_limit} to " + f"{new_fd_limit}: {err}.") + diff --git a/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst new file mode 100644 index 0000000000000..2528857caf9c0 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-21-17-22-26.bpo-43592.kHRsra.rst @@ -0,0 +1,3 @@ +:mod:`test.libregrtest` now raises the soft resource limit for the maximum +number of file descriptors when the default is too low for our test suite as +was often the case on macOS. From webhook-mailer at python.org Fri Oct 22 04:12:12 2021 From: webhook-mailer at python.org (encukou) Date: Fri, 22 Oct 2021 08:12:12 -0000 Subject: [Python-checkins] bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354) Message-ID: https://github.com/python/cpython/commit/276468dddb46c54980c782c09cdb53bd90755752 commit: 276468dddb46c54980c782c09cdb53bd90755752 branch: main author: Petr Viktorin committer: encukou date: 2021-10-22T10:12:06+02:00 summary: bpo-43795: Add a test for Stable ABI symbol availability using ctypes (GH-26354) This is a cross-platform check that the symbols are actually exported in the ABI, not e.g. hidden in a macro. Caveat: PyModule_Create2 & PyModule_FromDefAndSpec2 are skipped. These aren't exported on some of our buildbots. This is a bug (bpo-44133). This test now makes sure all the others don't regress. files: A Lib/test/test_stable_abi_ctypes.py M Tools/scripts/stable_abi.py diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py new file mode 100644 index 0000000000000..750aa18108327 --- /dev/null +++ b/Lib/test/test_stable_abi_ctypes.py @@ -0,0 +1,843 @@ + +# Generated by Tools/scripts/stable_abi.py + +"""Test that all symbols of the Stable ABI are accessible using ctypes +""" + +import unittest +from test.support.import_helper import import_module + +ctypes_test = import_module('ctypes') + +class TestStableABIAvailability(unittest.TestCase): + def test_available_symbols(self): + for symbol_name in SYMBOL_NAMES: + with self.subTest(symbol_name): + ctypes_test.pythonapi[symbol_name] + +SYMBOL_NAMES = ( + + "PyAIter_Check", + "PyArg_Parse", + "PyArg_ParseTuple", + "PyArg_ParseTupleAndKeywords", + "PyArg_UnpackTuple", + "PyArg_VaParse", + "PyArg_VaParseTupleAndKeywords", + "PyArg_ValidateKeywordArguments", + "PyBaseObject_Type", + "PyBool_FromLong", + "PyBool_Type", + "PyByteArrayIter_Type", + "PyByteArray_AsString", + "PyByteArray_Concat", + "PyByteArray_FromObject", + "PyByteArray_FromStringAndSize", + "PyByteArray_Resize", + "PyByteArray_Size", + "PyByteArray_Type", + "PyBytesIter_Type", + "PyBytes_AsString", + "PyBytes_AsStringAndSize", + "PyBytes_Concat", + "PyBytes_ConcatAndDel", + "PyBytes_DecodeEscape", + "PyBytes_FromFormat", + "PyBytes_FromFormatV", + "PyBytes_FromObject", + "PyBytes_FromString", + "PyBytes_FromStringAndSize", + "PyBytes_Repr", + "PyBytes_Size", + "PyBytes_Type", + "PyCFunction_Call", + "PyCFunction_GetFlags", + "PyCFunction_GetFunction", + "PyCFunction_GetSelf", + "PyCFunction_New", + "PyCFunction_NewEx", + "PyCFunction_Type", + "PyCMethod_New", + "PyCallIter_New", + "PyCallIter_Type", + "PyCallable_Check", + "PyCapsule_GetContext", + "PyCapsule_GetDestructor", + "PyCapsule_GetName", + "PyCapsule_GetPointer", + "PyCapsule_Import", + "PyCapsule_IsValid", + "PyCapsule_New", + "PyCapsule_SetContext", + "PyCapsule_SetDestructor", + "PyCapsule_SetName", + "PyCapsule_SetPointer", + "PyCapsule_Type", + "PyClassMethodDescr_Type", + "PyCodec_BackslashReplaceErrors", + "PyCodec_Decode", + "PyCodec_Decoder", + "PyCodec_Encode", + "PyCodec_Encoder", + "PyCodec_IgnoreErrors", + "PyCodec_IncrementalDecoder", + "PyCodec_IncrementalEncoder", + "PyCodec_KnownEncoding", + "PyCodec_LookupError", + "PyCodec_NameReplaceErrors", + "PyCodec_Register", + "PyCodec_RegisterError", + "PyCodec_ReplaceErrors", + "PyCodec_StreamReader", + "PyCodec_StreamWriter", + "PyCodec_StrictErrors", + "PyCodec_Unregister", + "PyCodec_XMLCharRefReplaceErrors", + "PyComplex_FromDoubles", + "PyComplex_ImagAsDouble", + "PyComplex_RealAsDouble", + "PyComplex_Type", + "PyDescr_NewClassMethod", + "PyDescr_NewGetSet", + "PyDescr_NewMember", + "PyDescr_NewMethod", + "PyDictItems_Type", + "PyDictIterItem_Type", + "PyDictIterKey_Type", + "PyDictIterValue_Type", + "PyDictKeys_Type", + "PyDictProxy_New", + "PyDictProxy_Type", + "PyDictRevIterItem_Type", + "PyDictRevIterKey_Type", + "PyDictRevIterValue_Type", + "PyDictValues_Type", + "PyDict_Clear", + "PyDict_Contains", + "PyDict_Copy", + "PyDict_DelItem", + "PyDict_DelItemString", + "PyDict_GetItem", + "PyDict_GetItemString", + "PyDict_GetItemWithError", + "PyDict_Items", + "PyDict_Keys", + "PyDict_Merge", + "PyDict_MergeFromSeq2", + "PyDict_New", + "PyDict_Next", + "PyDict_SetItem", + "PyDict_SetItemString", + "PyDict_Size", + "PyDict_Type", + "PyDict_Update", + "PyDict_Values", + "PyEllipsis_Type", + "PyEnum_Type", + "PyErr_BadArgument", + "PyErr_BadInternalCall", + "PyErr_CheckSignals", + "PyErr_Clear", + "PyErr_Display", + "PyErr_ExceptionMatches", + "PyErr_Fetch", + "PyErr_Format", + "PyErr_FormatV", + "PyErr_GetExcInfo", + "PyErr_GivenExceptionMatches", + "PyErr_NewException", + "PyErr_NewExceptionWithDoc", + "PyErr_NoMemory", + "PyErr_NormalizeException", + "PyErr_Occurred", + "PyErr_Print", + "PyErr_PrintEx", + "PyErr_ProgramText", + "PyErr_ResourceWarning", + "PyErr_Restore", + "PyErr_SetExcInfo", + "PyErr_SetFromErrno", + "PyErr_SetFromErrnoWithFilename", + "PyErr_SetFromErrnoWithFilenameObject", + "PyErr_SetFromErrnoWithFilenameObjects", + "PyErr_SetImportError", + "PyErr_SetImportErrorSubclass", + "PyErr_SetInterrupt", + "PyErr_SetInterruptEx", + "PyErr_SetNone", + "PyErr_SetObject", + "PyErr_SetString", + "PyErr_SyntaxLocation", + "PyErr_SyntaxLocationEx", + "PyErr_WarnEx", + "PyErr_WarnExplicit", + "PyErr_WarnFormat", + "PyErr_WriteUnraisable", + "PyEval_AcquireLock", + "PyEval_AcquireThread", + "PyEval_CallFunction", + "PyEval_CallMethod", + "PyEval_CallObjectWithKeywords", + "PyEval_EvalCode", + "PyEval_EvalCodeEx", + "PyEval_EvalFrame", + "PyEval_EvalFrameEx", + "PyEval_GetBuiltins", + "PyEval_GetFrame", + "PyEval_GetFuncDesc", + "PyEval_GetFuncName", + "PyEval_GetGlobals", + "PyEval_GetLocals", + "PyEval_InitThreads", + "PyEval_ReleaseLock", + "PyEval_ReleaseThread", + "PyEval_RestoreThread", + "PyEval_SaveThread", + "PyEval_ThreadsInitialized", + "PyExc_ArithmeticError", + "PyExc_AssertionError", + "PyExc_AttributeError", + "PyExc_BaseException", + "PyExc_BlockingIOError", + "PyExc_BrokenPipeError", + "PyExc_BufferError", + "PyExc_BytesWarning", + "PyExc_ChildProcessError", + "PyExc_ConnectionAbortedError", + "PyExc_ConnectionError", + "PyExc_ConnectionRefusedError", + "PyExc_ConnectionResetError", + "PyExc_DeprecationWarning", + "PyExc_EOFError", + "PyExc_EncodingWarning", + "PyExc_EnvironmentError", + "PyExc_Exception", + "PyExc_FileExistsError", + "PyExc_FileNotFoundError", + "PyExc_FloatingPointError", + "PyExc_FutureWarning", + "PyExc_GeneratorExit", + "PyExc_IOError", + "PyExc_ImportError", + "PyExc_ImportWarning", + "PyExc_IndentationError", + "PyExc_IndexError", + "PyExc_InterruptedError", + "PyExc_IsADirectoryError", + "PyExc_KeyError", + "PyExc_KeyboardInterrupt", + "PyExc_LookupError", + "PyExc_MemoryError", + "PyExc_ModuleNotFoundError", + "PyExc_NameError", + "PyExc_NotADirectoryError", + "PyExc_NotImplementedError", + "PyExc_OSError", + "PyExc_OverflowError", + "PyExc_PendingDeprecationWarning", + "PyExc_PermissionError", + "PyExc_ProcessLookupError", + "PyExc_RecursionError", + "PyExc_ReferenceError", + "PyExc_ResourceWarning", + "PyExc_RuntimeError", + "PyExc_RuntimeWarning", + "PyExc_StopAsyncIteration", + "PyExc_StopIteration", + "PyExc_SyntaxError", + "PyExc_SyntaxWarning", + "PyExc_SystemError", + "PyExc_SystemExit", + "PyExc_TabError", + "PyExc_TimeoutError", + "PyExc_TypeError", + "PyExc_UnboundLocalError", + "PyExc_UnicodeDecodeError", + "PyExc_UnicodeEncodeError", + "PyExc_UnicodeError", + "PyExc_UnicodeTranslateError", + "PyExc_UnicodeWarning", + "PyExc_UserWarning", + "PyExc_ValueError", + "PyExc_Warning", + "PyExc_ZeroDivisionError", + "PyExceptionClass_Name", + "PyException_GetCause", + "PyException_GetContext", + "PyException_GetTraceback", + "PyException_SetCause", + "PyException_SetContext", + "PyException_SetTraceback", + "PyFile_FromFd", + "PyFile_GetLine", + "PyFile_WriteObject", + "PyFile_WriteString", + "PyFilter_Type", + "PyFloat_AsDouble", + "PyFloat_FromDouble", + "PyFloat_FromString", + "PyFloat_GetInfo", + "PyFloat_GetMax", + "PyFloat_GetMin", + "PyFloat_Type", + "PyFrame_GetCode", + "PyFrame_GetLineNumber", + "PyFrozenSet_New", + "PyFrozenSet_Type", + "PyGC_Collect", + "PyGC_Disable", + "PyGC_Enable", + "PyGC_IsEnabled", + "PyGILState_Ensure", + "PyGILState_GetThisThreadState", + "PyGILState_Release", + "PyGetSetDescr_Type", + "PyImport_AddModule", + "PyImport_AddModuleObject", + "PyImport_AppendInittab", + "PyImport_ExecCodeModule", + "PyImport_ExecCodeModuleEx", + "PyImport_ExecCodeModuleObject", + "PyImport_ExecCodeModuleWithPathnames", + "PyImport_GetImporter", + "PyImport_GetMagicNumber", + "PyImport_GetMagicTag", + "PyImport_GetModule", + "PyImport_GetModuleDict", + "PyImport_Import", + "PyImport_ImportFrozenModule", + "PyImport_ImportFrozenModuleObject", + "PyImport_ImportModule", + "PyImport_ImportModuleLevel", + "PyImport_ImportModuleLevelObject", + "PyImport_ImportModuleNoBlock", + "PyImport_ReloadModule", + "PyIndex_Check", + "PyInterpreterState_Clear", + "PyInterpreterState_Delete", + "PyInterpreterState_Get", + "PyInterpreterState_GetDict", + "PyInterpreterState_GetID", + "PyInterpreterState_New", + "PyIter_Check", + "PyIter_Next", + "PyIter_Send", + "PyListIter_Type", + "PyListRevIter_Type", + "PyList_Append", + "PyList_AsTuple", + "PyList_GetItem", + "PyList_GetSlice", + "PyList_Insert", + "PyList_New", + "PyList_Reverse", + "PyList_SetItem", + "PyList_SetSlice", + "PyList_Size", + "PyList_Sort", + "PyList_Type", + "PyLongRangeIter_Type", + "PyLong_AsDouble", + "PyLong_AsLong", + "PyLong_AsLongAndOverflow", + "PyLong_AsLongLong", + "PyLong_AsLongLongAndOverflow", + "PyLong_AsSize_t", + "PyLong_AsSsize_t", + "PyLong_AsUnsignedLong", + "PyLong_AsUnsignedLongLong", + "PyLong_AsUnsignedLongLongMask", + "PyLong_AsUnsignedLongMask", + "PyLong_AsVoidPtr", + "PyLong_FromDouble", + "PyLong_FromLong", + "PyLong_FromLongLong", + "PyLong_FromSize_t", + "PyLong_FromSsize_t", + "PyLong_FromString", + "PyLong_FromUnsignedLong", + "PyLong_FromUnsignedLongLong", + "PyLong_FromVoidPtr", + "PyLong_GetInfo", + "PyLong_Type", + "PyMap_Type", + "PyMapping_Check", + "PyMapping_GetItemString", + "PyMapping_HasKey", + "PyMapping_HasKeyString", + "PyMapping_Items", + "PyMapping_Keys", + "PyMapping_Length", + "PyMapping_SetItemString", + "PyMapping_Size", + "PyMapping_Values", + "PyMarshal_ReadObjectFromString", + "PyMarshal_WriteObjectToString", + "PyMem_Calloc", + "PyMem_Free", + "PyMem_Malloc", + "PyMem_Realloc", + "PyMemberDescr_Type", + "PyMember_GetOne", + "PyMember_SetOne", + "PyMemoryView_FromMemory", + "PyMemoryView_FromObject", + "PyMemoryView_GetContiguous", + "PyMemoryView_Type", + "PyMethodDescr_Type", + "PyModuleDef_Init", + "PyModuleDef_Type", + "PyModule_AddFunctions", + "PyModule_AddIntConstant", + "PyModule_AddObject", + "PyModule_AddObjectRef", + "PyModule_AddStringConstant", + "PyModule_AddType", + "PyModule_ExecDef", + "PyModule_GetDef", + "PyModule_GetDict", + "PyModule_GetFilename", + "PyModule_GetFilenameObject", + "PyModule_GetName", + "PyModule_GetNameObject", + "PyModule_GetState", + "PyModule_New", + "PyModule_NewObject", + "PyModule_SetDocString", + "PyModule_Type", + "PyNumber_Absolute", + "PyNumber_Add", + "PyNumber_And", + "PyNumber_AsSsize_t", + "PyNumber_Check", + "PyNumber_Divmod", + "PyNumber_Float", + "PyNumber_FloorDivide", + "PyNumber_InPlaceAdd", + "PyNumber_InPlaceAnd", + "PyNumber_InPlaceFloorDivide", + "PyNumber_InPlaceLshift", + "PyNumber_InPlaceMatrixMultiply", + "PyNumber_InPlaceMultiply", + "PyNumber_InPlaceOr", + "PyNumber_InPlacePower", + "PyNumber_InPlaceRemainder", + "PyNumber_InPlaceRshift", + "PyNumber_InPlaceSubtract", + "PyNumber_InPlaceTrueDivide", + "PyNumber_InPlaceXor", + "PyNumber_Index", + "PyNumber_Invert", + "PyNumber_Long", + "PyNumber_Lshift", + "PyNumber_MatrixMultiply", + "PyNumber_Multiply", + "PyNumber_Negative", + "PyNumber_Or", + "PyNumber_Positive", + "PyNumber_Power", + "PyNumber_Remainder", + "PyNumber_Rshift", + "PyNumber_Subtract", + "PyNumber_ToBase", + "PyNumber_TrueDivide", + "PyNumber_Xor", + "PyOS_FSPath", + "PyOS_InputHook", + "PyOS_InterruptOccurred", + "PyOS_double_to_string", + "PyOS_getsig", + "PyOS_mystricmp", + "PyOS_mystrnicmp", + "PyOS_setsig", + "PyOS_snprintf", + "PyOS_string_to_double", + "PyOS_strtol", + "PyOS_strtoul", + "PyOS_vsnprintf", + "PyObject_ASCII", + "PyObject_AsCharBuffer", + "PyObject_AsFileDescriptor", + "PyObject_AsReadBuffer", + "PyObject_AsWriteBuffer", + "PyObject_Bytes", + "PyObject_Call", + "PyObject_CallFunction", + "PyObject_CallFunctionObjArgs", + "PyObject_CallMethod", + "PyObject_CallMethodObjArgs", + "PyObject_CallNoArgs", + "PyObject_CallObject", + "PyObject_Calloc", + "PyObject_CheckReadBuffer", + "PyObject_ClearWeakRefs", + "PyObject_DelItem", + "PyObject_DelItemString", + "PyObject_Dir", + "PyObject_Format", + "PyObject_Free", + "PyObject_GC_Del", + "PyObject_GC_IsFinalized", + "PyObject_GC_IsTracked", + "PyObject_GC_Track", + "PyObject_GC_UnTrack", + "PyObject_GenericGetAttr", + "PyObject_GenericGetDict", + "PyObject_GenericSetAttr", + "PyObject_GenericSetDict", + "PyObject_GetAIter", + "PyObject_GetAttr", + "PyObject_GetAttrString", + "PyObject_GetItem", + "PyObject_GetIter", + "PyObject_HasAttr", + "PyObject_HasAttrString", + "PyObject_Hash", + "PyObject_HashNotImplemented", + "PyObject_Init", + "PyObject_InitVar", + "PyObject_IsInstance", + "PyObject_IsSubclass", + "PyObject_IsTrue", + "PyObject_Length", + "PyObject_Malloc", + "PyObject_Not", + "PyObject_Realloc", + "PyObject_Repr", + "PyObject_RichCompare", + "PyObject_RichCompareBool", + "PyObject_SelfIter", + "PyObject_SetAttr", + "PyObject_SetAttrString", + "PyObject_SetItem", + "PyObject_Size", + "PyObject_Str", + "PyObject_Type", + "PyProperty_Type", + "PyRangeIter_Type", + "PyRange_Type", + "PyReversed_Type", + "PySeqIter_New", + "PySeqIter_Type", + "PySequence_Check", + "PySequence_Concat", + "PySequence_Contains", + "PySequence_Count", + "PySequence_DelItem", + "PySequence_DelSlice", + "PySequence_Fast", + "PySequence_GetItem", + "PySequence_GetSlice", + "PySequence_In", + "PySequence_InPlaceConcat", + "PySequence_InPlaceRepeat", + "PySequence_Index", + "PySequence_Length", + "PySequence_List", + "PySequence_Repeat", + "PySequence_SetItem", + "PySequence_SetSlice", + "PySequence_Size", + "PySequence_Tuple", + "PySetIter_Type", + "PySet_Add", + "PySet_Clear", + "PySet_Contains", + "PySet_Discard", + "PySet_New", + "PySet_Pop", + "PySet_Size", + "PySet_Type", + "PySlice_AdjustIndices", + "PySlice_GetIndices", + "PySlice_GetIndicesEx", + "PySlice_New", + "PySlice_Type", + "PySlice_Unpack", + "PyState_AddModule", + "PyState_FindModule", + "PyState_RemoveModule", + "PyStructSequence_GetItem", + "PyStructSequence_New", + "PyStructSequence_NewType", + "PyStructSequence_SetItem", + "PyStructSequence_UnnamedField", + "PySuper_Type", + "PySys_AddWarnOption", + "PySys_AddWarnOptionUnicode", + "PySys_AddXOption", + "PySys_FormatStderr", + "PySys_FormatStdout", + "PySys_GetObject", + "PySys_GetXOptions", + "PySys_HasWarnOptions", + "PySys_ResetWarnOptions", + "PySys_SetArgv", + "PySys_SetArgvEx", + "PySys_SetObject", + "PySys_SetPath", + "PySys_WriteStderr", + "PySys_WriteStdout", + "PyThreadState_Clear", + "PyThreadState_Delete", + "PyThreadState_DeleteCurrent", + "PyThreadState_Get", + "PyThreadState_GetDict", + "PyThreadState_GetFrame", + "PyThreadState_GetID", + "PyThreadState_GetInterpreter", + "PyThreadState_New", + "PyThreadState_SetAsyncExc", + "PyThreadState_Swap", + "PyThread_GetInfo", + "PyThread_ReInitTLS", + "PyThread_acquire_lock", + "PyThread_acquire_lock_timed", + "PyThread_allocate_lock", + "PyThread_create_key", + "PyThread_delete_key", + "PyThread_delete_key_value", + "PyThread_exit_thread", + "PyThread_free_lock", + "PyThread_get_key_value", + "PyThread_get_stacksize", + "PyThread_get_thread_ident", + "PyThread_get_thread_native_id", + "PyThread_init_thread", + "PyThread_release_lock", + "PyThread_set_key_value", + "PyThread_set_stacksize", + "PyThread_start_new_thread", + "PyThread_tss_alloc", + "PyThread_tss_create", + "PyThread_tss_delete", + "PyThread_tss_free", + "PyThread_tss_get", + "PyThread_tss_is_created", + "PyThread_tss_set", + "PyTraceBack_Here", + "PyTraceBack_Print", + "PyTraceBack_Type", + "PyTupleIter_Type", + "PyTuple_GetItem", + "PyTuple_GetSlice", + "PyTuple_New", + "PyTuple_Pack", + "PyTuple_SetItem", + "PyTuple_Size", + "PyTuple_Type", + "PyType_ClearCache", + "PyType_FromModuleAndSpec", + "PyType_FromSpec", + "PyType_FromSpecWithBases", + "PyType_GenericAlloc", + "PyType_GenericNew", + "PyType_GetFlags", + "PyType_GetModule", + "PyType_GetModuleState", + "PyType_GetName", + "PyType_GetQualName", + "PyType_GetSlot", + "PyType_IsSubtype", + "PyType_Modified", + "PyType_Ready", + "PyType_Type", + "PyUnicodeDecodeError_Create", + "PyUnicodeDecodeError_GetEncoding", + "PyUnicodeDecodeError_GetEnd", + "PyUnicodeDecodeError_GetObject", + "PyUnicodeDecodeError_GetReason", + "PyUnicodeDecodeError_GetStart", + "PyUnicodeDecodeError_SetEnd", + "PyUnicodeDecodeError_SetReason", + "PyUnicodeDecodeError_SetStart", + "PyUnicodeEncodeError_GetEncoding", + "PyUnicodeEncodeError_GetEnd", + "PyUnicodeEncodeError_GetObject", + "PyUnicodeEncodeError_GetReason", + "PyUnicodeEncodeError_GetStart", + "PyUnicodeEncodeError_SetEnd", + "PyUnicodeEncodeError_SetReason", + "PyUnicodeEncodeError_SetStart", + "PyUnicodeIter_Type", + "PyUnicodeTranslateError_GetEnd", + "PyUnicodeTranslateError_GetObject", + "PyUnicodeTranslateError_GetReason", + "PyUnicodeTranslateError_GetStart", + "PyUnicodeTranslateError_SetEnd", + "PyUnicodeTranslateError_SetReason", + "PyUnicodeTranslateError_SetStart", + "PyUnicode_Append", + "PyUnicode_AppendAndDel", + "PyUnicode_AsASCIIString", + "PyUnicode_AsCharmapString", + "PyUnicode_AsDecodedObject", + "PyUnicode_AsDecodedUnicode", + "PyUnicode_AsEncodedObject", + "PyUnicode_AsEncodedString", + "PyUnicode_AsEncodedUnicode", + "PyUnicode_AsLatin1String", + "PyUnicode_AsRawUnicodeEscapeString", + "PyUnicode_AsUCS4", + "PyUnicode_AsUCS4Copy", + "PyUnicode_AsUTF16String", + "PyUnicode_AsUTF32String", + "PyUnicode_AsUTF8AndSize", + "PyUnicode_AsUTF8String", + "PyUnicode_AsUnicodeEscapeString", + "PyUnicode_AsWideChar", + "PyUnicode_AsWideCharString", + "PyUnicode_BuildEncodingMap", + "PyUnicode_Compare", + "PyUnicode_CompareWithASCIIString", + "PyUnicode_Concat", + "PyUnicode_Contains", + "PyUnicode_Count", + "PyUnicode_Decode", + "PyUnicode_DecodeASCII", + "PyUnicode_DecodeCharmap", + "PyUnicode_DecodeFSDefault", + "PyUnicode_DecodeFSDefaultAndSize", + "PyUnicode_DecodeLatin1", + "PyUnicode_DecodeLocale", + "PyUnicode_DecodeLocaleAndSize", + "PyUnicode_DecodeRawUnicodeEscape", + "PyUnicode_DecodeUTF16", + "PyUnicode_DecodeUTF16Stateful", + "PyUnicode_DecodeUTF32", + "PyUnicode_DecodeUTF32Stateful", + "PyUnicode_DecodeUTF7", + "PyUnicode_DecodeUTF7Stateful", + "PyUnicode_DecodeUTF8", + "PyUnicode_DecodeUTF8Stateful", + "PyUnicode_DecodeUnicodeEscape", + "PyUnicode_EncodeFSDefault", + "PyUnicode_EncodeLocale", + "PyUnicode_FSConverter", + "PyUnicode_FSDecoder", + "PyUnicode_Find", + "PyUnicode_FindChar", + "PyUnicode_Format", + "PyUnicode_FromEncodedObject", + "PyUnicode_FromFormat", + "PyUnicode_FromFormatV", + "PyUnicode_FromObject", + "PyUnicode_FromOrdinal", + "PyUnicode_FromString", + "PyUnicode_FromStringAndSize", + "PyUnicode_FromWideChar", + "PyUnicode_GetDefaultEncoding", + "PyUnicode_GetLength", + "PyUnicode_GetSize", + "PyUnicode_InternFromString", + "PyUnicode_InternImmortal", + "PyUnicode_InternInPlace", + "PyUnicode_IsIdentifier", + "PyUnicode_Join", + "PyUnicode_Partition", + "PyUnicode_RPartition", + "PyUnicode_RSplit", + "PyUnicode_ReadChar", + "PyUnicode_Replace", + "PyUnicode_Resize", + "PyUnicode_RichCompare", + "PyUnicode_Split", + "PyUnicode_Splitlines", + "PyUnicode_Substring", + "PyUnicode_Tailmatch", + "PyUnicode_Translate", + "PyUnicode_Type", + "PyUnicode_WriteChar", + "PyWeakref_GetObject", + "PyWeakref_NewProxy", + "PyWeakref_NewRef", + "PyWrapperDescr_Type", + "PyWrapper_New", + "PyZip_Type", + "Py_AddPendingCall", + "Py_AtExit", + "Py_BuildValue", + "Py_BytesMain", + "Py_CompileString", + "Py_DecRef", + "Py_DecodeLocale", + "Py_EncodeLocale", + "Py_EndInterpreter", + "Py_EnterRecursiveCall", + "Py_Exit", + "Py_FatalError", + "Py_FileSystemDefaultEncodeErrors", + "Py_FileSystemDefaultEncoding", + "Py_Finalize", + "Py_FinalizeEx", + "Py_GenericAlias", + "Py_GenericAliasType", + "Py_GetArgcArgv", + "Py_GetBuildInfo", + "Py_GetCompiler", + "Py_GetCopyright", + "Py_GetExecPrefix", + "Py_GetPath", + "Py_GetPlatform", + "Py_GetPrefix", + "Py_GetProgramFullPath", + "Py_GetProgramName", + "Py_GetPythonHome", + "Py_GetRecursionLimit", + "Py_GetVersion", + "Py_HasFileSystemDefaultEncoding", + "Py_IncRef", + "Py_Initialize", + "Py_InitializeEx", + "Py_Is", + "Py_IsFalse", + "Py_IsInitialized", + "Py_IsNone", + "Py_IsTrue", + "Py_LeaveRecursiveCall", + "Py_Main", + "Py_MakePendingCalls", + "Py_NewInterpreter", + "Py_NewRef", + "Py_ReprEnter", + "Py_ReprLeave", + "Py_SetPath", + "Py_SetProgramName", + "Py_SetPythonHome", + "Py_SetRecursionLimit", + "Py_UTF8Mode", + "Py_VaBuildValue", + "Py_XNewRef", + "_PyArg_ParseTupleAndKeywords_SizeT", + "_PyArg_ParseTuple_SizeT", + "_PyArg_Parse_SizeT", + "_PyArg_VaParseTupleAndKeywords_SizeT", + "_PyArg_VaParse_SizeT", + "_PyErr_BadInternalCall", + "_PyObject_CallFunction_SizeT", + "_PyObject_CallMethod_SizeT", + "_PyObject_GC_Malloc", + "_PyObject_GC_New", + "_PyObject_GC_NewVar", + "_PyObject_GC_Resize", + "_PyObject_New", + "_PyObject_NewVar", + "_PyState_AddModule", + "_PyThreadState_Init", + "_PyThreadState_Prealloc", + "_PyWeakref_CallableProxyType", + "_PyWeakref_ProxyType", + "_PyWeakref_RefType", + "_Py_BuildValue_SizeT", + "_Py_CheckRecursiveCall", + "_Py_Dealloc", + "_Py_DecRef", + "_Py_EllipsisObject", + "_Py_FalseStruct", + "_Py_IncRef", + "_Py_NoneStruct", + "_Py_NotImplementedStruct", + "_Py_SwappedOp", + "_Py_TrueStruct", + "_Py_VaBuildValue_SizeT", +) diff --git a/Tools/scripts/stable_abi.py b/Tools/scripts/stable_abi.py index 6cb310e5a31d5..0179f3c12fcbd 100755 --- a/Tools/scripts/stable_abi.py +++ b/Tools/scripts/stable_abi.py @@ -258,6 +258,44 @@ def gen_doc_annotations(manifest, args, outfile): 'added': item.added, 'ifdef_note': ifdef_note}) + at generator("ctypes_test", 'Lib/test/test_stable_abi_ctypes.py') +def gen_ctypes_test(manifest, args, outfile): + """Generate/check the ctypes-based test for exported symbols""" + write = partial(print, file=outfile) + write(textwrap.dedent(''' + # Generated by Tools/scripts/stable_abi.py + + """Test that all symbols of the Stable ABI are accessible using ctypes + """ + + import unittest + from test.support.import_helper import import_module + + ctypes_test = import_module('ctypes') + + class TestStableABIAvailability(unittest.TestCase): + def test_available_symbols(self): + for symbol_name in SYMBOL_NAMES: + with self.subTest(symbol_name): + ctypes_test.pythonapi[symbol_name] + + SYMBOL_NAMES = ( + ''')) + items = manifest.select( + {'function', 'data'}, + include_abi_only=True, + ifdef=set()) + for item in items: + if item.name in ( + # Some symbols aren't exported on all platforms. + # This is a bug: https://bugs.python.org/issue44133 + 'PyModule_Create2', 'PyModule_FromDefAndSpec2', + ): + continue + write(f' "{item.name}",') + write(")") + + def generate_or_check(manifest, args, path, func): """Generate/check a file with a single generator From webhook-mailer at python.org Fri Oct 22 04:52:29 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 08:52:29 -0000 Subject: [Python-checkins] bpo-45562: Only show debug output from the parser in debug builds (GH-29140) Message-ID: https://github.com/python/cpython/commit/86dfb55d2e091cf633dbd7aabcd49d96fb1f9d81 commit: 86dfb55d2e091cf633dbd7aabcd49d96fb1f9d81 branch: main author: Pablo Galindo Salgado committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T01:52:24-07:00 summary: bpo-45562: Only show debug output from the parser in debug builds (GH-29140) files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index ae3874b09de48..789a10b28fd1b 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1043,11 +1043,13 @@ tok_nextc(struct tok_state *tok) else { rc = tok_underflow_file(tok); } +#if defined(Py_DEBUG) if (Py_DebugFlag) { printf("line[%d] = ", tok->lineno); print_escape(stdout, tok->cur, tok->inp - tok->cur); printf(" tok->done = %d\n", tok->done); } +#endif if (!rc) { tok->cur = tok->inp; return EOF; From webhook-mailer at python.org Fri Oct 22 06:14:53 2021 From: webhook-mailer at python.org (pablogsal) Date: Fri, 22 Oct 2021 10:14:53 -0000 Subject: [Python-checkins] bpo-45562: Only show debug output from the parser in debug builds (GH-29140) (#29149) Message-ID: https://github.com/python/cpython/commit/ae78ffdc9399802621eabcd1668e44a91ec5f45e commit: ae78ffdc9399802621eabcd1668e44a91ec5f45e branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: pablogsal date: 2021-10-22T11:14:47+01:00 summary: bpo-45562: Only show debug output from the parser in debug builds (GH-29140) (#29149) (cherry picked from commit 86dfb55d2e091cf633dbd7aabcd49d96fb1f9d81) Co-authored-by: Pablo Galindo Salgado Co-authored-by: Pablo Galindo Salgado files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index a076d625bbf51..53c10282ce86a 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1042,11 +1042,13 @@ tok_nextc(struct tok_state *tok) else { rc = tok_underflow_file(tok); } +#if defined(Py_DEBUG) if (Py_DebugFlag) { printf("line[%d] = ", tok->lineno); print_escape(stdout, tok->cur, tok->inp - tok->cur); printf(" tok->done = %d\n", tok->done); } +#endif if (!rc) { tok->cur = tok->inp; return EOF; From webhook-mailer at python.org Fri Oct 22 07:08:54 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 11:08:54 -0000 Subject: [Python-checkins] bpo-44959: Add fallback to extension modules with '.sl' suffix on HP-UX (GH-27857) Message-ID: https://github.com/python/cpython/commit/aa8c3446c085175e65e736b0d2e32719c00004d2 commit: aa8c3446c085175e65e736b0d2e32719c00004d2 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T04:08:50-07:00 summary: bpo-44959: Add fallback to extension modules with '.sl' suffix on HP-UX (GH-27857) (cherry picked from commit 2396fa6537d79554ac694dbd2b0b30eeb3476c80) Co-authored-by: Florin Sp?tar files: A Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst M Python/dynload_hpux.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst new file mode 100644 index 0000000000000..02e11ae94e430 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-08-08-29-41.bpo-44959.OSwwPf.rst @@ -0,0 +1 @@ +Added fallback to extension modules with '.sl' suffix on HP-UX \ No newline at end of file diff --git a/Python/dynload_hpux.c b/Python/dynload_hpux.c index e36d608c6dca4..a53373038ed85 100644 --- a/Python/dynload_hpux.c +++ b/Python/dynload_hpux.c @@ -13,7 +13,7 @@ #define FUNCNAME_PATTERN "%.20s_%.200s" #endif -const char *_PyImport_DynLoadFiletab[] = {SHLIB_EXT, NULL}; +const char *_PyImport_DynLoadFiletab[] = {SHLIB_EXT, ".sl", NULL}; dl_funcptr _PyImport_FindSharedFuncptr(const char *prefix, const char *shortname, From webhook-mailer at python.org Fri Oct 22 09:36:38 2021 From: webhook-mailer at python.org (tiran) Date: Fri, 22 Oct 2021 13:36:38 -0000 Subject: [Python-checkins] bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157) Message-ID: https://github.com/python/cpython/commit/03e9f5dc751b8c441a85f428abc3f432ffe46345 commit: 03e9f5dc751b8c441a85f428abc3f432ffe46345 branch: main author: Christian Heimes committer: tiran date: 2021-10-22T15:36:28+02:00 summary: bpo-43974: Move Py_BUILD_CORE_MODULE into module code (GH-29157) setup.py no longer defines Py_BUILD_CORE_MODULE. Instead every module defines the macro before #include "Python.h" unless Py_BUILD_CORE_BUILTIN is already defined. Py_BUILD_CORE_BUILTIN is defined for every module that is built by Modules/Setup. The PR also simplifies Modules/Setup. Makefile and makesetup already define Py_BUILD_CORE_BUILTIN and include Modules/internal for us. Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-22-14-00-44.bpo-43974.HHZtbx.rst M Modules/Setup M Modules/_abc.c M Modules/_asynciomodule.c M Modules/_blake2/blake2b_impl.c M Modules/_blake2/blake2module.c M Modules/_blake2/blake2s_impl.c M Modules/_ctypes/_ctypes.c M Modules/_ctypes/callbacks.c M Modules/_ctypes/cfield.c M Modules/_ctypes/stgdict.c M Modules/_cursesmodule.c M Modules/_datetimemodule.c M Modules/_decimal/_decimal.c M Modules/_hashopenssl.c M Modules/_heapqmodule.c M Modules/_json.c M Modules/_lsprof.c M Modules/_math.c M Modules/_pickle.c M Modules/_posixsubprocess.c M Modules/_queuemodule.c M Modules/_randommodule.c M Modules/_sha3/sha3module.c M Modules/_struct.c M Modules/_testinternalcapi.c M Modules/_testmultiphase.c M Modules/_xxsubinterpretersmodule.c M Modules/_zoneinfo.c M Modules/arraymodule.c M Modules/binascii.c M Modules/cmathmodule.c M Modules/mathmodule.c M Modules/md5module.c M Modules/sha1module.c M Modules/sha256module.c M Modules/sha512module.c M Modules/unicodedata.c M PCbuild/pyproject.props M setup.py diff --git a/Misc/NEWS.d/next/Build/2021-10-22-14-00-44.bpo-43974.HHZtbx.rst b/Misc/NEWS.d/next/Build/2021-10-22-14-00-44.bpo-43974.HHZtbx.rst new file mode 100644 index 0000000000000..728a095dc03af --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-22-14-00-44.bpo-43974.HHZtbx.rst @@ -0,0 +1,2 @@ +``setup.py`` no longer defines ``Py_BUILD_CORE_MODULE``. Instead every +module, that uses the internal API, defines the macro. diff --git a/Modules/Setup b/Modules/Setup index a4440dfb6fc9d..a5b1a7b9c14af 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -99,26 +99,26 @@ PYTHONPATH=$(COREPYTHONPATH) # cannot be built as shared! _collections _collectionsmodule.c -_abc -DPy_BUILD_CORE_BUILTIN _abc.c +_abc _abc.c _codecs _codecsmodule.c -_functools -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _functoolsmodule.c -_io -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c -_locale -DPy_BUILD_CORE_BUILTIN _localemodule.c # -lintl -_operator -DPy_BUILD_CORE_BUILTIN _operator.c -_signal -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal signalmodule.c -_sre -DPy_BUILD_CORE_BUILTIN _sre.c +_functools _functoolsmodule.c +_io -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c +_locale _localemodule.c # -lintl +_operator _operator.c +_signal signalmodule.c +_sre _sre.c _stat _stat.c _symtable symtablemodule.c # setup.py can't track the .h file that _symtable depends on. -_thread -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _threadmodule.c +_thread _threadmodule.c _tracemalloc _tracemalloc.c # See bpo-35053 as to why this is built in. _weakref _weakref.c atexit atexitmodule.c errno errnomodule.c faulthandler faulthandler.c itertools itertoolsmodule.c -posix -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal posixmodule.c +posix posixmodule.c pwd pwdmodule.c -time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c +time timemodule.c # --- @@ -139,7 +139,7 @@ time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c #_asyncio _asynciomodule.c #_bisect _bisectmodule.c -#_blake2 -DPy_BUILD_CORE_BUILTIN _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c +#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c #_codecs_cn cjkcodecs/_codecs_cn.c #_codecs_hk cjkcodecs/_codecs_hk.c #_codecs_iso2022 cjkcodecs/_codecs_iso2022.c @@ -150,31 +150,31 @@ time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c #_csv _csv.c #_datetime _datetimemodule.c #_elementtree -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat _elementtree.c -#_heapq -DPy_BUILD_CORE_MODULE _heapqmodule.c -#_json -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal _json.c +#_heapq _heapqmodule.c +#_json _json.c #_lsprof _lsprof.c rotatingtree.c -#_md5 -DPy_BUILD_CORE_BUILTIN md5module.c +#_md5 md5module.c #_multibytecodec cjkcodecs/multibytecodec.c #_opcode _opcode.c -#_pickle -DPy_BUILD_CORE_MODULE _pickle.c -#_posixsubprocess -DPy_BUILD_CORE_BUILTIN _posixsubprocess.c -#_queue -DPy_BUILD_CORE_MODULE _queuemodule.c -#_random -DPy_BUILD_CORE_MODULE _randommodule.c -#_sha1 -DPy_BUILD_CORE_BUILTIN sha1module.c -#_sha256 -DPy_BUILD_CORE_BUILTIN sha256module.c -#_sha512 -DPy_BUILD_CORE_BUILTIN sha512module.c -#_sha3 -DPy_BUILD_CORE_BUILTIN _sha3/sha3module.c +#_pickle _pickle.c +#_posixsubprocess _posixsubprocess.c +#_queue _queuemodule.c +#_random _randommodule.c +#_sha1 sha1module.c +#_sha256 sha256module.c +#_sha512 sha512module.c +#_sha3 _sha3/sha3module.c #_statistics _statisticsmodule.c -#_struct -DPy_BUILD_CORE_MODULE _struct.c +#_struct _struct.c #_typing _typingmodule.c -#_zoneinfo -DPy_BUILD_CORE_MODULE _zoneinfo.c -#array -DPy_BUILD_CORE_MODULE arraymodule.c +#_zoneinfo _zoneinfo.c +#array arraymodule.c #audioop audioop.c -#binascii -DPy_BUILD_CORE_MODULE binascii.c -#cmath -DPy_BUILD_CORE_MODULE cmathmodule.c _math.c # -lm -#math -DPy_BUILD_CORE_MODULE mathmodule.c _math.c # -lm +#binascii binascii.c +#cmath cmathmodule.c _math.c # -lm +#math mathmodule.c _math.c # -lm #pyexpat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -#unicodedata -DPy_BUILD_CORE_BUILTIN unicodedata.c +#unicodedata unicodedata.c # Modules with some UNIX dependencies -- on by default: # (If you have a really backward UNIX, select and socket may not be @@ -286,7 +286,7 @@ time -DPy_BUILD_CORE_BUILTIN -I$(srcdir)/Include/internal timemodule.c # provided by the ncurses library. e.g. on Linux, link with -lncurses # instead of -lcurses). -#_curses -DPy_BUILD_CORE_MODULE -lcurses -ltermcap _cursesmodule.c +#_curses -lcurses -ltermcap _cursesmodule.c # Wrapper for the panel library that's part of ncurses and SYSV curses. #_curses_panel -lpanel -lncurses _curses_panel.c @@ -305,7 +305,7 @@ xxsubtype xxsubtype.c # Required for the test suite to pass! #_testbuffer _testbuffer.c #_testcapi _testcapimodule.c # CANNOT be statically compiled! #_testimportmultiple _testimportmultiple.c -#_testinternalcapi -DPy_BUILD_CORE_MODULE -I$(srcdir)/Include/internal _testinternalcapi.c +#_testinternalcapi _testinternalcapi.c #_testmultiphase _testmultiphase.c diff --git a/Modules/_abc.c b/Modules/_abc.c index 8aa68359039e7..b7465c379dddf 100644 --- a/Modules/_abc.c +++ b/Modules/_abc.c @@ -1,4 +1,7 @@ /* ABCMeta implementation */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "pycore_moduleobject.h" // _PyModule_GetState() diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index adc5ff9f796cc..8386a50d55826 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_pyerrors.h" // _PyErr_ClearExcState() #include "pycore_pystate.h" // _PyThreadState_GET() diff --git a/Modules/_blake2/blake2b_impl.c b/Modules/_blake2/blake2b_impl.c index b16324e8f7aeb..e1421dd8ff8b8 100644 --- a/Modules/_blake2/blake2b_impl.c +++ b/Modules/_blake2/blake2b_impl.c @@ -13,6 +13,10 @@ * The blake2s_impl.c is autogenerated from blake2b_impl.c. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_strhex.h" // _Py_strhex() diff --git a/Modules/_blake2/blake2module.c b/Modules/_blake2/blake2module.c index 631de2cc0abc7..3b6bba277a313 100644 --- a/Modules/_blake2/blake2module.c +++ b/Modules/_blake2/blake2module.c @@ -8,6 +8,10 @@ * any warranty. http://creativecommons.org/publicdomain/zero/1.0/ */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "impl/blake2.h" diff --git a/Modules/_blake2/blake2s_impl.c b/Modules/_blake2/blake2s_impl.c index 6b31a363ea27b..763c0178e6bcd 100644 --- a/Modules/_blake2/blake2s_impl.c +++ b/Modules/_blake2/blake2s_impl.c @@ -13,6 +13,10 @@ * The blake2s_impl.c is autogenerated from blake2s_impl.c. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_strhex.h" // _Py_strhex() diff --git a/Modules/_ctypes/_ctypes.c b/Modules/_ctypes/_ctypes.c index f8940fdbed242..96078c7726d59 100644 --- a/Modules/_ctypes/_ctypes.c +++ b/Modules/_ctypes/_ctypes.c @@ -98,6 +98,9 @@ bytes(cdata) * PyCField_Type * */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #define PY_SSIZE_T_CLEAN diff --git a/Modules/_ctypes/callbacks.c b/Modules/_ctypes/callbacks.c index c24f04ce3a0f8..0f7789a973e8f 100644 --- a/Modules/_ctypes/callbacks.c +++ b/Modules/_ctypes/callbacks.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" // windows.h must be included before pycore internal headers #ifdef MS_WIN32 diff --git a/Modules/_ctypes/cfield.c b/Modules/_ctypes/cfield.c index 2cfd657028aca..515b280182098 100644 --- a/Modules/_ctypes/cfield.c +++ b/Modules/_ctypes/cfield.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" // windows.h must be included before pycore internal headers #ifdef MS_WIN32 diff --git a/Modules/_ctypes/stgdict.c b/Modules/_ctypes/stgdict.c index 43669d7152a7e..6c1917b18d450 100644 --- a/Modules/_ctypes/stgdict.c +++ b/Modules/_ctypes/stgdict.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" // windows.h must be included before pycore internal headers #ifdef MS_WIN32 diff --git a/Modules/_cursesmodule.c b/Modules/_cursesmodule.c index 511073f2ac137..3770a032e977d 100644 --- a/Modules/_cursesmodule.c +++ b/Modules/_cursesmodule.c @@ -100,6 +100,10 @@ static const char PyCursesVersion[] = "2.2"; /* Includes */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" diff --git a/Modules/_datetimemodule.c b/Modules/_datetimemodule.c index e54a01b44c5ba..67441eba28f7f 100644 --- a/Modules/_datetimemodule.c +++ b/Modules/_datetimemodule.c @@ -7,6 +7,10 @@ * the capsule are defined below */ #define _PY_DATETIME_IMPL +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_long.h" // _PyLong_GetOne() #include "pycore_object.h" // _PyObject_Init() diff --git a/Modules/_decimal/_decimal.c b/Modules/_decimal/_decimal.c index 237edd5191fd9..7fc7315603e7a 100644 --- a/Modules/_decimal/_decimal.c +++ b/Modules/_decimal/_decimal.c @@ -25,6 +25,9 @@ * SUCH DAMAGE. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include #include "pycore_pystate.h" // _PyThreadState_GET() diff --git a/Modules/_hashopenssl.c b/Modules/_hashopenssl.c index b4ba60baaac93..12491917832b6 100644 --- a/Modules/_hashopenssl.c +++ b/Modules/_hashopenssl.c @@ -18,6 +18,10 @@ #endif #define OPENSSL_NO_DEPRECATED 1 +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" diff --git a/Modules/_heapqmodule.c b/Modules/_heapqmodule.c index 20468c28f2423..3dbaaa0a0da1d 100644 --- a/Modules/_heapqmodule.c +++ b/Modules/_heapqmodule.c @@ -6,6 +6,10 @@ annotated by Fran?ois Pinard, and converted to C by Raymond Hettinger. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_list.h" // _PyList_ITEMS() diff --git a/Modules/_json.c b/Modules/_json.c index 6f68c1f7f9b71..1c9c50665d466 100644 --- a/Modules/_json.c +++ b/Modules/_json.c @@ -4,8 +4,8 @@ * and as an extension module (Py_BUILD_CORE_MODULE define) on other * platforms. */ -#if !defined(Py_BUILD_CORE_BUILTIN) && !defined(Py_BUILD_CORE_MODULE) -# error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined" +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 #endif #include "Python.h" diff --git a/Modules/_lsprof.c b/Modules/_lsprof.c index 2e27afcea1b79..ff499aacbab37 100644 --- a/Modules/_lsprof.c +++ b/Modules/_lsprof.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_pystate.h" // _PyThreadState_GET() diff --git a/Modules/_math.c b/Modules/_math.c index 68e3a2346925d..c1936a1088a24 100644 --- a/Modules/_math.c +++ b/Modules/_math.c @@ -1,6 +1,10 @@ /* Definitions of some C99 math library functions, for those platforms that don't implement these functions already. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include #include "_math.h" diff --git a/Modules/_pickle.c b/Modules/_pickle.c index 53ab57111967c..0d9e57aa90d22 100644 --- a/Modules/_pickle.c +++ b/Modules/_pickle.c @@ -4,8 +4,8 @@ * and as an extension module (Py_BUILD_CORE_MODULE define) on other * platforms. */ -#if !defined(Py_BUILD_CORE_BUILTIN) && !defined(Py_BUILD_CORE_MODULE) -# error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined" +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 #endif #include "Python.h" diff --git a/Modules/_posixsubprocess.c b/Modules/_posixsubprocess.c index 63207de8b9137..de599f8c970e3 100644 --- a/Modules/_posixsubprocess.c +++ b/Modules/_posixsubprocess.c @@ -1,4 +1,8 @@ /* Authors: Gregory P. Smith & Jeffrey Yasskin */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_fileutils.h" #if defined(HAVE_PIPE2) && !defined(_GNU_SOURCE) diff --git a/Modules/_queuemodule.c b/Modules/_queuemodule.c index eb61349b76581..413387fecc4cd 100644 --- a/Modules/_queuemodule.c +++ b/Modules/_queuemodule.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_moduleobject.h" // _PyModule_GetState() #include "structmember.h" // PyMemberDef diff --git a/Modules/_randommodule.c b/Modules/_randommodule.c index 65d41f4e8e80c..5243d5a05e290 100644 --- a/Modules/_randommodule.c +++ b/Modules/_randommodule.c @@ -66,6 +66,10 @@ /* ---------------------------------------------------------------*/ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_moduleobject.h" // _PyModule_GetState() #ifdef HAVE_PROCESS_H diff --git a/Modules/_sha3/sha3module.c b/Modules/_sha3/sha3module.c index a033c4e452545..bfa96616df0c9 100644 --- a/Modules/_sha3/sha3module.c +++ b/Modules/_sha3/sha3module.c @@ -15,6 +15,10 @@ * */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_strhex.h" // _Py_strhex() #include "../hashlib.h" diff --git a/Modules/_struct.c b/Modules/_struct.c index a8003a90b2682..210dbdc752f4c 100644 --- a/Modules/_struct.c +++ b/Modules/_struct.c @@ -3,6 +3,10 @@ /* New version supporting byte order, alignment and size options, character strings, and unsigned numbers */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index 3ba939651a417..1ca06069e1119 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -2,8 +2,8 @@ * C Extension module to test Python internal C APIs (Include/internal). */ -#if !defined(Py_BUILD_CORE_BUILTIN) && !defined(Py_BUILD_CORE_MODULE) -# error "Py_BUILD_CORE_BUILTIN or Py_BUILD_CORE_MODULE must be defined" +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 #endif /* Always enable assertions */ diff --git a/Modules/_testmultiphase.c b/Modules/_testmultiphase.c index 2d25e16bd4d39..ee69c42336170 100644 --- a/Modules/_testmultiphase.c +++ b/Modules/_testmultiphase.c @@ -1,6 +1,9 @@ /* Testing module for multi-phase initialization of extension modules (PEP 489) */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "pycore_namespace.h" // _PyNamespace_New() diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index b5c0a63219114..3f683768fcc28 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -1,6 +1,9 @@ /* interpreters module */ /* low-level access to interpreter primitives */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "frameobject.h" diff --git a/Modules/_zoneinfo.c b/Modules/_zoneinfo.c index 04fa09422b213..cac347071f91d 100644 --- a/Modules/_zoneinfo.c +++ b/Modules/_zoneinfo.c @@ -1,3 +1,7 @@ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_long.h" // _PyLong_GetOne() #include "structmember.h" diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c index 5e57fe116059d..030ede57351e9 100644 --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -3,6 +3,10 @@ /* An array is a uniform list -- all items have the same type. The item type is restricted to simple C types like int or float */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" #include "pycore_floatobject.h" // _PyFloat_Unpack4() diff --git a/Modules/binascii.c b/Modules/binascii.c index 7037d34dbe2dd..fec0d82a39cdd 100644 --- a/Modules/binascii.c +++ b/Modules/binascii.c @@ -53,6 +53,10 @@ ** Brandon Long, September 2001. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" diff --git a/Modules/cmathmodule.c b/Modules/cmathmodule.c index 0f22049a17084..0e0489c5fe2d7 100644 --- a/Modules/cmathmodule.c +++ b/Modules/cmathmodule.c @@ -2,6 +2,10 @@ /* much code borrowed from mathmodule.c */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_dtoa.h" #include "_math.h" diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 4fac0cc29e4e9..6c12a4e70ddf2 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -52,6 +52,10 @@ raised for division by zero and mod by zero. returned. */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #include "Python.h" #include "pycore_bitutils.h" // _Py_bit_length() #include "pycore_call.h" // _PyObject_CallNoArgs() diff --git a/Modules/md5module.c b/Modules/md5module.c index 4d03f6b844b33..48b11e0779f87 100644 --- a/Modules/md5module.c +++ b/Modules/md5module.c @@ -15,6 +15,9 @@ */ /* MD5 objects */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "hashlib.h" diff --git a/Modules/sha1module.c b/Modules/sha1module.c index 153bc12a8bd74..9153557fbde74 100644 --- a/Modules/sha1module.c +++ b/Modules/sha1module.c @@ -15,6 +15,9 @@ */ /* SHA1 objects */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "hashlib.h" diff --git a/Modules/sha256module.c b/Modules/sha256module.c index 5858071db4e13..17ee86683b7a8 100644 --- a/Modules/sha256module.c +++ b/Modules/sha256module.c @@ -15,6 +15,9 @@ */ /* SHA objects */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "pycore_bitutils.h" // _Py_bswap32() diff --git a/Modules/sha512module.c b/Modules/sha512module.c index e50b69be4617d..bf4408b455f2c 100644 --- a/Modules/sha512module.c +++ b/Modules/sha512module.c @@ -15,6 +15,9 @@ */ /* SHA objects */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif #include "Python.h" #include "pycore_bitutils.h" // _Py_bswap64() diff --git a/Modules/unicodedata.c b/Modules/unicodedata.c index e863e53bfa9f1..bdbddcf10b778 100644 --- a/Modules/unicodedata.c +++ b/Modules/unicodedata.c @@ -12,6 +12,10 @@ ------------------------------------------------------------------------ */ +#ifndef Py_BUILD_CORE_BUILTIN +# define Py_BUILD_CORE_MODULE 1 +#endif + #define PY_SSIZE_T_CLEAN #include "Python.h" diff --git a/PCbuild/pyproject.props b/PCbuild/pyproject.props index d492b71dfbaa3..bbcabb5cdb405 100644 --- a/PCbuild/pyproject.props +++ b/PCbuild/pyproject.props @@ -25,7 +25,6 @@ <_DebugPreprocessorDefinition Condition="$(Configuration) == 'Debug'">_DEBUG; <_PlatformPreprocessorDefinition>_WIN32; <_PlatformPreprocessorDefinition Condition="$(Platform) == 'x64'">_WIN64;_M_X64; - <_PydPreprocessorDefinition Condition="$(TargetExt) == '.pyd'">Py_BUILD_CORE_MODULE; <_Py3NamePreprocessorDefinition>PY3_DLLNAME=L"$(Py3DllName)"; diff --git a/setup.py b/setup.py index 5428cbde1cc9b..03dce63984ef3 100644 --- a/setup.py +++ b/setup.py @@ -899,8 +899,7 @@ def detect_simple_extensions(self): # # array objects - self.add(Extension('array', ['arraymodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('array', ['arraymodule.c'])) # Context Variables self.add(Extension('_contextvars', ['_contextvarsmodule.c'])) @@ -909,14 +908,12 @@ def detect_simple_extensions(self): # math library functions, e.g. sin() self.add(Extension('math', ['mathmodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], extra_objects=[shared_math], depends=['_math.h', shared_math], libraries=['m'])) # complex math library functions self.add(Extension('cmath', ['cmathmodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], extra_objects=[shared_math], depends=['_math.h', shared_math], libraries=['m'])) @@ -933,44 +930,33 @@ def detect_simple_extensions(self): # libm is needed by delta_new() that uses round() and by accum() that # uses modf(). self.add(Extension('_datetime', ['_datetimemodule.c'], - libraries=['m'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + libraries=['m'])) # zoneinfo module - self.add(Extension('_zoneinfo', ['_zoneinfo.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_zoneinfo', ['_zoneinfo.c'])) # random number generator implemented in C - self.add(Extension("_random", ["_randommodule.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_random", ["_randommodule.c"])) # bisect self.add(Extension("_bisect", ["_bisectmodule.c"])) # heapq - self.add(Extension("_heapq", ["_heapqmodule.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_heapq", ["_heapqmodule.c"])) # C-optimized pickle replacement - self.add(Extension("_pickle", ["_pickle.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_pickle", ["_pickle.c"])) # _json speedups - self.add(Extension("_json", ["_json.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_json", ["_json.c"])) # profiler (_lsprof is for cProfile.py) - self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_lsprof', ['_lsprof.c', 'rotatingtree.c'])) # static Unicode character database self.add(Extension('unicodedata', ['unicodedata.c'], - depends=['unicodedata_db.h', 'unicodename_db.h'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + depends=['unicodedata_db.h', 'unicodename_db.h'])) # _opcode module self.add(Extension('_opcode', ['_opcode.c'])) # asyncio speedups - self.add(Extension("_asyncio", ["_asynciomodule.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_asyncio", ["_asynciomodule.c"])) # _abc speedups - self.add(Extension("_abc", ["_abc.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_abc", ["_abc.c"])) # _queue module - self.add(Extension("_queue", ["_queuemodule.c"], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension("_queue", ["_queuemodule.c"])) # _statistics module self.add(Extension("_statistics", ["_statisticsmodule.c"])) # _typing module @@ -1012,8 +998,7 @@ def detect_simple_extensions(self): self.add(Extension('syslog', ['syslogmodule.c'])) # Python interface to subinterpreter C-API. - self.add(Extension('_xxsubinterpreters', ['_xxsubinterpretersmodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_xxsubinterpreters', ['_xxsubinterpretersmodule.c'])) # # Here ends the simple stuff. From here on, modules need certain @@ -1036,8 +1021,7 @@ def detect_simple_extensions(self): self.add(Extension('_csv', ['_csv.c'])) # POSIX subprocess module helper. - self.add(Extension('_posixsubprocess', ['_posixsubprocess.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_posixsubprocess', ['_posixsubprocess.c'])) def detect_test_extensions(self): # Python C API test module @@ -1045,8 +1029,7 @@ def detect_test_extensions(self): depends=['testcapi_long.h'])) # Python Internal C API test module - self.add(Extension('_testinternalcapi', ['_testinternalcapi.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_testinternalcapi', ['_testinternalcapi.c'])) # Python PEP-3118 (buffer protocol) test module self.add(Extension('_testbuffer', ['_testbuffer.c'])) @@ -1055,8 +1038,7 @@ def detect_test_extensions(self): self.add(Extension('_testimportmultiple', ['_testimportmultiple.c'])) # Test multi-phase extension module init (PEP 489) - self.add(Extension('_testmultiphase', ['_testmultiphase.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])) + self.add(Extension('_testmultiphase', ['_testmultiphase.c'])) # Fuzz tests. self.add(Extension('_xxtestfuzz', @@ -1187,7 +1169,6 @@ def detect_readline_curses(self): if curses_library.startswith('ncurses'): curses_libs = [curses_library] self.add(Extension('_curses', ['_cursesmodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], include_dirs=curses_includes, define_macros=curses_defines, libraries=curses_libs)) @@ -1202,7 +1183,6 @@ def detect_readline_curses(self): curses_libs = ['curses'] self.add(Extension('_curses', ['_cursesmodule.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], define_macros=curses_defines, libraries=curses_libs)) else: @@ -1722,7 +1702,7 @@ def detect_compress_exts(self): # Helper module for various ascii-encoders. Uses zlib for an optimized # crc32 if we have it. Otherwise binascii uses its own. - extra_compile_args = ['-DPy_BUILD_CORE_MODULE'] + extra_compile_args = [] if have_zlib: extra_compile_args.append('-DUSE_ZLIB_CRC32') libraries = ['z'] @@ -2230,7 +2210,7 @@ def detect_ctypes(self): self.use_system_libffi = '--with-system-ffi' in sysconfig.get_config_var("CONFIG_ARGS") include_dirs = [] - extra_compile_args = ['-DPy_BUILD_CORE_MODULE'] + extra_compile_args = [] extra_link_args = [] sources = ['_ctypes/_ctypes.c', '_ctypes/callbacks.c', @@ -2324,7 +2304,7 @@ def detect_ctypes(self): def detect_decimal(self): # Stefan Krah's _decimal module - extra_compile_args = ['-DPy_BUILD_CORE_MODULE'] + extra_compile_args = [] undef_macros = [] if '--with-system-libmpdec' in sysconfig.get_config_var("CONFIG_ARGS"): include_dirs = [] @@ -2482,7 +2462,6 @@ def split_var(name, sep): library_dirs=openssl_libdirs, libraries=openssl_libs, runtime_library_dirs=runtime_library_dirs, - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], ) # This static linking is NOT OFFICIALLY SUPPORTED. @@ -2545,28 +2524,24 @@ def detect_hash_builtins(self): self.add(Extension( '_sha256', ['sha256module.c'], depends=['hashlib.h'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "sha512" in configured: self.add(Extension( '_sha512', ['sha512module.c'], depends=['hashlib.h'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "md5" in configured: self.add(Extension( '_md5', ['md5module.c'], depends=['hashlib.h'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "sha1" in configured: self.add(Extension( '_sha1', ['sha1module.c'], depends=['hashlib.h'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "blake2" in configured: @@ -2582,7 +2557,6 @@ def detect_hash_builtins(self): '_blake2/blake2s_impl.c' ], depends=blake2_deps, - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) if "sha3" in configured: @@ -2594,7 +2568,6 @@ def detect_hash_builtins(self): '_sha3', ['_sha3/sha3module.c'], depends=sha3_deps, - extra_compile_args=['-DPy_BUILD_CORE_MODULE'], )) def detect_nis(self): @@ -2750,8 +2723,7 @@ class DummyProcess: 'install_lib': PyBuildInstallLib}, # The struct module is defined here, because build_ext won't be # called unless there's at least one extension module defined. - ext_modules=[Extension('_struct', ['_struct.c'], - extra_compile_args=['-DPy_BUILD_CORE_MODULE'])], + ext_modules=[Extension('_struct', ['_struct.c'])], # If you change the scripts installed here, you also need to # check the PyBuildScripts command above, and change the links From webhook-mailer at python.org Fri Oct 22 10:15:02 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 22 Oct 2021 14:15:02 -0000 Subject: [Python-checkins] bpo-45502: Fix test_shelve (GH-29003) Message-ID: https://github.com/python/cpython/commit/b781cc3bfce7c052728b06aad9f1a467cced289d commit: b781cc3bfce7c052728b06aad9f1a467cced289d branch: main author: Serhiy Storchaka committer: ambv date: 2021-10-22T16:14:58+02:00 summary: bpo-45502: Fix test_shelve (GH-29003) Run test_shelve with all underlying dbm implementations and pickle protocols. Also make test_shelve discoverable. files: M Lib/test/test_shelve.py diff --git a/Lib/test/test_shelve.py b/Lib/test/test_shelve.py index 03c03472775a0..b9eb007c82796 100644 --- a/Lib/test/test_shelve.py +++ b/Lib/test/test_shelve.py @@ -1,4 +1,5 @@ import unittest +import dbm import shelve import glob import pickle @@ -44,12 +45,8 @@ def copy(self): class TestCase(unittest.TestCase): - - fn = "shelftemp.db" - - def tearDown(self): - for f in glob.glob(self.fn+"*"): - os_helper.unlink(f) + dirname = os_helper.TESTFN + fn = os.path.join(os_helper.TESTFN, "shelftemp.db") def test_close(self): d1 = {} @@ -67,6 +64,8 @@ def test_close(self): self.fail('Closed shelf should not find a key') def test_open_template(self, filename=None, protocol=None): + os.mkdir(self.dirname) + self.addCleanup(os_helper.rmtree, self.dirname) s = shelve.open(filename=filename if filename is not None else self.fn, protocol=protocol) try: @@ -168,63 +167,52 @@ def test_default_protocol(self): with shelve.Shelf({}) as s: self.assertEqual(s._protocol, pickle.DEFAULT_PROTOCOL) -from test import mapping_tests -class TestShelveBase(mapping_tests.BasicTestMappingProtocol): - fn = "shelftemp.db" - counter = 0 - def __init__(self, *args, **kw): - self._db = [] - mapping_tests.BasicTestMappingProtocol.__init__(self, *args, **kw) +class TestShelveBase: type2test = shelve.Shelf + def _reference(self): return {"key1":"value1", "key2":2, "key3":(1,2,3)} + + +class TestShelveInMemBase(TestShelveBase): def _empty_mapping(self): - if self._in_mem: - x= shelve.Shelf(byteskeydict(), **self._args) - else: - self.counter+=1 - x= shelve.open(self.fn+str(self.counter), **self._args) - self._db.append(x) + return shelve.Shelf(byteskeydict(), **self._args) + + +class TestShelveFileBase(TestShelveBase): + counter = 0 + + def _empty_mapping(self): + self.counter += 1 + x = shelve.open(self.base_path + str(self.counter), **self._args) + self.addCleanup(x.close) return x - def tearDown(self): - for db in self._db: - db.close() - self._db = [] - if not self._in_mem: - for f in glob.glob(self.fn+"*"): - os_helper.unlink(f) - -class TestAsciiFileShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = False -class TestBinaryFileShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = False -class TestProto2FileShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = False -class TestAsciiMemShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = True -class TestBinaryMemShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = True -class TestProto2MemShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = True - -def test_main(): - for module in dbm_iterator(): - support.run_unittest( - TestAsciiFileShelve, - TestBinaryFileShelve, - TestProto2FileShelve, - TestAsciiMemShelve, - TestBinaryMemShelve, - TestProto2MemShelve, - TestCase - ) + + def setUp(self): + dirname = os_helper.TESTFN + os.mkdir(dirname) + self.addCleanup(os_helper.rmtree, dirname) + self.base_path = os.path.join(dirname, "shelftemp.db") + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) + dbm._defaultmod = self.dbm_mod + + +from test import mapping_tests + +for proto in range(pickle.HIGHEST_PROTOCOL + 1): + bases = (TestShelveInMemBase, mapping_tests.BasicTestMappingProtocol) + name = f'TestProto{proto}MemShelve' + globals()[name] = type(name, bases, + {'_args': {'protocol': proto}}) + bases = (TestShelveFileBase, mapping_tests.BasicTestMappingProtocol) + for dbm_mod in dbm_iterator(): + assert dbm_mod.__name__.startswith('dbm.') + suffix = dbm_mod.__name__[4:] + name = f'TestProto{proto}File_{suffix}Shelve' + globals()[name] = type(name, bases, + {'dbm_mod': dbm_mod, '_args': {'protocol': proto}}) + if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 22 12:28:27 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 16:28:27 -0000 Subject: [Python-checkins] bpo-45570: Simplify setup macros for pyexpat (GH-29159) Message-ID: https://github.com/python/cpython/commit/ec93721e0066c4cbe40085188a9bf0952aa935ef commit: ec93721e0066c4cbe40085188a9bf0952aa935ef branch: main author: Christian Heimes committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T09:28:23-07:00 summary: bpo-45570: Simplify setup macros for pyexpat (GH-29159) * ``HAVE_EXPAT_CONFIG_H`` is not used by our code and not used by system-wide expat header files * ``USE_PYEXPAT_CAPI`` is no longer used by our code * ``XML_POOR_ENTROPY`` should be defined in expat_config.h Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-22-14-45-40.bpo-45570.61gM2A.rst M Modules/Setup M Modules/expat/expat_config.h M PCbuild/_elementtree.vcxproj M PCbuild/pyexpat.vcxproj M setup.py diff --git a/Misc/NEWS.d/next/Build/2021-10-22-14-45-40.bpo-45570.61gM2A.rst b/Misc/NEWS.d/next/Build/2021-10-22-14-45-40.bpo-45570.61gM2A.rst new file mode 100644 index 0000000000000..7de50f911bf0d --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-22-14-45-40.bpo-45570.61gM2A.rst @@ -0,0 +1 @@ +:mod:`pyexpat` and :mod:`_elementtree` no longer define obsolete macros ``HAVE_EXPAT_CONFIG_H`` and ``USE_PYEXPAT_CAPI``. ``XML_POOR_ENTROPY`` is now defined in ``expat_config.h``. diff --git a/Modules/Setup b/Modules/Setup index a5b1a7b9c14af..d06d733fc9cbb 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -149,7 +149,7 @@ time timemodule.c #_contextvars _contextvarsmodule.c #_csv _csv.c #_datetime _datetimemodule.c -#_elementtree -DHAVE_EXPAT_CONFIG_H -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat _elementtree.c +#_elementtree -I$(srcdir)/Modules/expat _elementtree.c #_heapq _heapqmodule.c #_json _json.c #_lsprof _lsprof.c rotatingtree.c @@ -173,7 +173,7 @@ time timemodule.c #binascii binascii.c #cmath cmathmodule.c _math.c # -lm #math mathmodule.c _math.c # -lm -#pyexpat -DHAVE_EXPAT_CONFIG_H -DXML_POOR_ENTROPY -DUSE_PYEXPAT_CAPI -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c +#pyexpat -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c #unicodedata unicodedata.c # Modules with some UNIX dependencies -- on by default: diff --git a/Modules/expat/expat_config.h b/Modules/expat/expat_config.h index afbedd011f660..6671f7b689ba9 100644 --- a/Modules/expat/expat_config.h +++ b/Modules/expat/expat_config.h @@ -18,4 +18,8 @@ #define XML_DTD 1 #define XML_CONTEXT_BYTES 1024 +// bpo-30947: Python uses best available entropy sources to +// call XML_SetHashSalt(), expat entropy sources are not needed +#define XML_POOR_ENTROPY 1 + #endif /* EXPAT_CONFIG_H */ diff --git a/PCbuild/_elementtree.vcxproj b/PCbuild/_elementtree.vcxproj index 4a125b243b780..8da5244bac0cb 100644 --- a/PCbuild/_elementtree.vcxproj +++ b/PCbuild/_elementtree.vcxproj @@ -94,7 +94,7 @@ ..\Modules\expat;%(AdditionalIncludeDirectories) - _CRT_SECURE_NO_WARNINGS;USE_PYEXPAT_CAPI;XML_STATIC;%(PreprocessorDefinitions) + _CRT_SECURE_NO_WARNINGS;XML_STATIC;%(PreprocessorDefinitions) diff --git a/PCbuild/pyexpat.vcxproj b/PCbuild/pyexpat.vcxproj index b2d9f5d57d497..001f8afd89b9e 100644 --- a/PCbuild/pyexpat.vcxproj +++ b/PCbuild/pyexpat.vcxproj @@ -91,7 +91,7 @@ $(PySourcePath)Modules\expat;%(AdditionalIncludeDirectories) - _CRT_SECURE_NO_WARNINGS;PYEXPAT_EXPORTS;HAVE_EXPAT_H;XML_STATIC;%(PreprocessorDefinitions) + _CRT_SECURE_NO_WARNINGS;PYEXPAT_EXPORTS;XML_STATIC;%(PreprocessorDefinitions) diff --git a/setup.py b/setup.py index 03dce63984ef3..4bef3cb52f2b2 100644 --- a/setup.py +++ b/setup.py @@ -1748,19 +1748,12 @@ def detect_expat_elementtree(self): # if '--with-system-expat' in sysconfig.get_config_var("CONFIG_ARGS"): expat_inc = [] - define_macros = [] extra_compile_args = [] expat_lib = ['expat'] expat_sources = [] expat_depends = [] else: expat_inc = [os.path.join(self.srcdir, 'Modules', 'expat')] - define_macros = [ - ('HAVE_EXPAT_CONFIG_H', '1'), - # bpo-30947: Python uses best available entropy sources to - # call XML_SetHashSalt(), expat entropy sources are not needed - ('XML_POOR_ENTROPY', '1'), - ] extra_compile_args = [] # bpo-44394: libexpat uses isnan() of math.h and needs linkage # against the libm @@ -1788,7 +1781,6 @@ def detect_expat_elementtree(self): extra_compile_args.append('-Wno-unreachable-code') self.add(Extension('pyexpat', - define_macros=define_macros, extra_compile_args=extra_compile_args, include_dirs=expat_inc, libraries=expat_lib, @@ -1799,9 +1791,7 @@ def detect_expat_elementtree(self): # uses expat (via the CAPI hook in pyexpat). if os.path.isfile(os.path.join(self.srcdir, 'Modules', '_elementtree.c')): - define_macros.append(('USE_PYEXPAT_CAPI', None)) self.add(Extension('_elementtree', - define_macros=define_macros, include_dirs=expat_inc, libraries=expat_lib, sources=['_elementtree.c'], From webhook-mailer at python.org Fri Oct 22 14:03:08 2021 From: webhook-mailer at python.org (tiran) Date: Fri, 22 Oct 2021 18:03:08 -0000 Subject: [Python-checkins] bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) Message-ID: https://github.com/python/cpython/commit/f6e8b80d20159596cf641305bad3a833bedd2f4f commit: f6e8b80d20159596cf641305bad3a833bedd2f4f branch: main author: Christian Heimes committer: tiran date: 2021-10-22T20:02:54+02:00 summary: bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) files: A Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst M Modules/makesetup diff --git a/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst new file mode 100644 index 0000000000000..f2042d11e24b7 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst @@ -0,0 +1,2 @@ +``Modules/Setup`` now use ``PY_CFLAGS_NODIST`` instead of ``PY_CFLAGS`` to +compile shared modules. diff --git a/Modules/makesetup b/Modules/makesetup index fefe3fd129ee3..1a767838c92be 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -231,7 +231,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | *) src='$(srcdir)/'"$srcdir/$src";; esac case $doconfig in - no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS) \$(PY_CPPFLAGS)";; + no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS_NODIST) \$(PY_CPPFLAGS)";; *) cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac From webhook-mailer at python.org Fri Oct 22 14:22:21 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 18:22:21 -0000 Subject: [Python-checkins] bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) Message-ID: https://github.com/python/cpython/commit/19903085c3ad7a17c8047e1556c700f2eb109931 commit: 19903085c3ad7a17c8047e1556c700f2eb109931 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T11:22:01-07:00 summary: bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) (cherry picked from commit f6e8b80d20159596cf641305bad3a833bedd2f4f) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst M Modules/makesetup diff --git a/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst new file mode 100644 index 0000000000000..f2042d11e24b7 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst @@ -0,0 +1,2 @@ +``Modules/Setup`` now use ``PY_CFLAGS_NODIST`` instead of ``PY_CFLAGS`` to +compile shared modules. diff --git a/Modules/makesetup b/Modules/makesetup index fefe3fd129ee3..1a767838c92be 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -231,7 +231,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | *) src='$(srcdir)/'"$srcdir/$src";; esac case $doconfig in - no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS) \$(PY_CPPFLAGS)";; + no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS_NODIST) \$(PY_CPPFLAGS)";; *) cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac From webhook-mailer at python.org Fri Oct 22 14:30:02 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 18:30:02 -0000 Subject: [Python-checkins] bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) Message-ID: https://github.com/python/cpython/commit/269bf56e3d352c415ec38f93017ba8e291ddb18a commit: 269bf56e3d352c415ec38f93017ba8e291ddb18a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T11:29:52-07:00 summary: bpo-45571: use PY_CFLAGS_NODIST for shared Modules/Setup (GH-29161) (cherry picked from commit f6e8b80d20159596cf641305bad3a833bedd2f4f) Co-authored-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst M Modules/makesetup diff --git a/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst new file mode 100644 index 0000000000000..f2042d11e24b7 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-22-15-28-29.bpo-45571.yY8NsJ.rst @@ -0,0 +1,2 @@ +``Modules/Setup`` now use ``PY_CFLAGS_NODIST`` instead of ``PY_CFLAGS`` to +compile shared modules. diff --git a/Modules/makesetup b/Modules/makesetup index fefe3fd129ee3..1a767838c92be 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -231,7 +231,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | *) src='$(srcdir)/'"$srcdir/$src";; esac case $doconfig in - no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS) \$(PY_CPPFLAGS)";; + no) cc="$cc \$(CCSHARED) \$(PY_CFLAGS_NODIST) \$(PY_CPPFLAGS)";; *) cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac From webhook-mailer at python.org Fri Oct 22 17:24:17 2021 From: webhook-mailer at python.org (gpshead) Date: Fri, 22 Oct 2021 21:24:17 -0000 Subject: [Python-checkins] bpo-30570: Use Py_EnterRecursiveCall() in issubclass() (GH-29048) Message-ID: https://github.com/python/cpython/commit/423fa1c1817abfa8c3d1bc308ddbbd8f28b69d68 commit: 423fa1c1817abfa8c3d1bc308ddbbd8f28b69d68 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: gpshead date: 2021-10-22T14:24:08-07:00 summary: bpo-30570: Use Py_EnterRecursiveCall() in issubclass() (GH-29048) * Use Py_EnterRecursiveCall() in issubclass() Reviewed-by: Gregory P. Smith [Google] files: A Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst M Lib/test/test_isinstance.py M Objects/abstract.c diff --git a/Lib/test/test_isinstance.py b/Lib/test/test_isinstance.py index 109c3f84a5c42..6ab44be9a26ba 100644 --- a/Lib/test/test_isinstance.py +++ b/Lib/test/test_isinstance.py @@ -313,6 +313,36 @@ def __bases__(self): self.assertRaises(RecursionError, issubclass, int, X()) self.assertRaises(RecursionError, isinstance, 1, X()) + def test_infinite_recursion_via_bases_tuple(self): + """Regression test for bpo-30570.""" + class Failure(object): + def __getattr__(self, attr): + return (self, None) + + with self.assertRaises(RecursionError): + issubclass(Failure(), int) + + def test_infinite_cycle_in_bases(self): + """Regression test for bpo-30570.""" + class X: + @property + def __bases__(self): + return (self, self, self) + self.assertRaises(RecursionError, issubclass, X(), int) + + def test_infinitely_many_bases(self): + """Regression test for bpo-30570.""" + class X: + def __getattr__(self, attr): + self.assertEqual(attr, "__bases__") + class A: + pass + class B: + pass + A.__getattr__ = B.__getattr__ = X.__getattr__ + return (A(), B()) + self.assertRaises(RecursionError, issubclass, X(), int) + def blowstack(fxn, arg, compare_to): # Make sure that calling isinstance with a deeply nested tuple for its diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst new file mode 100644 index 0000000000000..d9ab60cd08f56 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst @@ -0,0 +1 @@ +Fixed a crash in ``issubclass()`` from infinite recursion when searching pathological ``__bases__`` tuples. \ No newline at end of file diff --git a/Objects/abstract.c b/Objects/abstract.c index 6f7b94600e278..6227ad5a18bb9 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2557,14 +2557,22 @@ abstract_issubclass(PyObject *derived, PyObject *cls) derived = PyTuple_GET_ITEM(bases, 0); continue; } - for (i = 0; i < n; i++) { - r = abstract_issubclass(PyTuple_GET_ITEM(bases, i), cls); - if (r != 0) - break; - } + break; + } + assert(n >= 2); + if (Py_EnterRecursiveCall(" in __issubclass__")) { Py_DECREF(bases); - return r; + return -1; } + for (i = 0; i < n; i++) { + r = abstract_issubclass(PyTuple_GET_ITEM(bases, i), cls); + if (r != 0) { + break; + } + } + Py_LeaveRecursiveCall(); + Py_DECREF(bases); + return r; } static int From webhook-mailer at python.org Fri Oct 22 17:47:04 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 21:47:04 -0000 Subject: [Python-checkins] bpo-30570: Use Py_EnterRecursiveCall() in issubclass() (GH-29048) Message-ID: https://github.com/python/cpython/commit/f812fef2f8f12441ce559335645433c8124e7db5 commit: f812fef2f8f12441ce559335645433c8124e7db5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T14:46:56-07:00 summary: bpo-30570: Use Py_EnterRecursiveCall() in issubclass() (GH-29048) * Use Py_EnterRecursiveCall() in issubclass() Reviewed-by: Gregory P. Smith [Google] (cherry picked from commit 423fa1c1817abfa8c3d1bc308ddbbd8f28b69d68) Co-authored-by: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> files: A Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst M Lib/test/test_isinstance.py M Objects/abstract.c diff --git a/Lib/test/test_isinstance.py b/Lib/test/test_isinstance.py index 109c3f84a5c42..6ab44be9a26ba 100644 --- a/Lib/test/test_isinstance.py +++ b/Lib/test/test_isinstance.py @@ -313,6 +313,36 @@ def __bases__(self): self.assertRaises(RecursionError, issubclass, int, X()) self.assertRaises(RecursionError, isinstance, 1, X()) + def test_infinite_recursion_via_bases_tuple(self): + """Regression test for bpo-30570.""" + class Failure(object): + def __getattr__(self, attr): + return (self, None) + + with self.assertRaises(RecursionError): + issubclass(Failure(), int) + + def test_infinite_cycle_in_bases(self): + """Regression test for bpo-30570.""" + class X: + @property + def __bases__(self): + return (self, self, self) + self.assertRaises(RecursionError, issubclass, X(), int) + + def test_infinitely_many_bases(self): + """Regression test for bpo-30570.""" + class X: + def __getattr__(self, attr): + self.assertEqual(attr, "__bases__") + class A: + pass + class B: + pass + A.__getattr__ = B.__getattr__ = X.__getattr__ + return (A(), B()) + self.assertRaises(RecursionError, issubclass, X(), int) + def blowstack(fxn, arg, compare_to): # Make sure that calling isinstance with a deeply nested tuple for its diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst new file mode 100644 index 0000000000000..d9ab60cd08f56 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-19-01-04-08.bpo-30570._G30Ms.rst @@ -0,0 +1 @@ +Fixed a crash in ``issubclass()`` from infinite recursion when searching pathological ``__bases__`` tuples. \ No newline at end of file diff --git a/Objects/abstract.c b/Objects/abstract.c index c056c6698850c..3ac85544defbd 100644 --- a/Objects/abstract.c +++ b/Objects/abstract.c @@ -2557,14 +2557,22 @@ abstract_issubclass(PyObject *derived, PyObject *cls) derived = PyTuple_GET_ITEM(bases, 0); continue; } - for (i = 0; i < n; i++) { - r = abstract_issubclass(PyTuple_GET_ITEM(bases, i), cls); - if (r != 0) - break; - } + break; + } + assert(n >= 2); + if (Py_EnterRecursiveCall(" in __issubclass__")) { Py_DECREF(bases); - return r; + return -1; } + for (i = 0; i < n; i++) { + r = abstract_issubclass(PyTuple_GET_ITEM(bases, i), cls); + if (r != 0) { + break; + } + } + Py_LeaveRecursiveCall(); + Py_DECREF(bases); + return r; } static int From webhook-mailer at python.org Fri Oct 22 17:57:33 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 22 Oct 2021 21:57:33 -0000 Subject: [Python-checkins] bpo-45574: fix warning about `print_escape` being unused (GH-29172) Message-ID: https://github.com/python/cpython/commit/4bc5473a42c5eae0928430930b897209492e849d commit: 4bc5473a42c5eae0928430930b897209492e849d branch: main author: Nikita Sobolev committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-22T14:57:24-07:00 summary: bpo-45574: fix warning about `print_escape` being unused (GH-29172) It used to be like this: ?????? ?????? 2021-10-22 ? 23 07 40 Quick `grep` tells that it is just used in one place under `Py_DEBUG`: https://github.com/python/cpython/blame/f6e8b80d20159596cf641305bad3a833bedd2f4f/Parser/tokenizer.c#L1047-L1051 ?????? ?????? 2021-10-22 ? 23 08 09 I am not sure, but it also looks like a private thing, it should not affect other users. Automerge-Triggered-By: GH:pablogsal files: A Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst M Parser/tokenizer.c diff --git a/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst b/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst new file mode 100644 index 0000000000000..b404d24473960 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst @@ -0,0 +1 @@ +Fix warning about ``print_escape`` being unused. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 789a10b28fd1b..705da00463d1a 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -995,6 +995,7 @@ tok_underflow_file(struct tok_state *tok) { return tok->done == E_OK; } +#if defined(Py_DEBUG) static void print_escape(FILE *f, const char *s, Py_ssize_t size) { @@ -1021,6 +1022,7 @@ print_escape(FILE *f, const char *s, Py_ssize_t size) } putc('"', f); } +#endif /* Get next char, updating state; error code goes into tok->done */ From webhook-mailer at python.org Fri Oct 22 19:13:51 2021 From: webhook-mailer at python.org (iritkatriel) Date: Fri, 22 Oct 2021 23:13:51 -0000 Subject: [Python-checkins] bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes (GH-28569) Message-ID: https://github.com/python/cpython/commit/f30ad65dbf3c6b1b5eec14dc954d65ef32327857 commit: f30ad65dbf3c6b1b5eec14dc954d65ef32327857 branch: main author: Irit Katriel <1055913+iritkatriel at users.noreply.github.com> committer: iritkatriel <1055913+iritkatriel at users.noreply.github.com> date: 2021-10-23T00:13:46+01:00 summary: bpo-45292: [PEP 654] add the ExceptionGroup and BaseExceptionGroup classes (GH-28569) files: A Lib/test/test_exception_group.py A Misc/NEWS.d/next/Core and Builtins/2021-09-26-18-18-50.bpo-45292.aX5HVr.rst M Doc/data/stable_abi.dat M Include/cpython/pyerrors.h M Include/internal/pycore_interp.h M Include/internal/pycore_pylifecycle.h M Include/pyerrors.h M Lib/test/exception_hierarchy.txt M Lib/test/test_descr.py M Lib/test/test_doctest.py M Lib/test/test_pickle.py M Lib/test/test_stable_abi_ctypes.py M Misc/stable_abi.txt M Objects/exceptions.c M PC/python3dll.c M Python/pylifecycle.c diff --git a/Doc/data/stable_abi.dat b/Doc/data/stable_abi.dat index 46ee321b660c3..64a0a2a247cd2 100644 --- a/Doc/data/stable_abi.dat +++ b/Doc/data/stable_abi.dat @@ -189,6 +189,7 @@ var,PyExc_ArithmeticError,3.2, var,PyExc_AssertionError,3.2, var,PyExc_AttributeError,3.2, var,PyExc_BaseException,3.2, +var,PyExc_BaseExceptionGroup,3.11, var,PyExc_BlockingIOError,3.7, var,PyExc_BrokenPipeError,3.7, var,PyExc_BufferError,3.2, diff --git a/Include/cpython/pyerrors.h b/Include/cpython/pyerrors.h index a3ec5afdb7c78..28ab565dde423 100644 --- a/Include/cpython/pyerrors.h +++ b/Include/cpython/pyerrors.h @@ -14,6 +14,12 @@ typedef struct { PyException_HEAD } PyBaseExceptionObject; +typedef struct { + PyException_HEAD + PyObject *msg; + PyObject *excs; +} PyBaseExceptionGroupObject; + typedef struct { PyException_HEAD PyObject *msg; diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index 64ac3abe00fa0..c16f0a4b5e643 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -205,6 +205,8 @@ struct _Py_exc_state { PyObject *errnomap; PyBaseExceptionObject *memerrors_freelist; int memerrors_numfree; + // The ExceptionGroup type + PyObject *PyExc_ExceptionGroup; }; diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h index 4f12fef8d6546..53b94748b32e9 100644 --- a/Include/internal/pycore_pylifecycle.h +++ b/Include/internal/pycore_pylifecycle.h @@ -93,6 +93,7 @@ extern void _PyAsyncGen_Fini(PyInterpreterState *interp); extern int _PySignal_Init(int install_signal_handlers); extern void _PySignal_Fini(void); +extern void _PyExc_ClearExceptionGroupType(PyInterpreterState *interp); extern void _PyExc_Fini(PyInterpreterState *interp); extern void _PyImport_Fini(void); extern void _PyImport_Fini2(void); diff --git a/Include/pyerrors.h b/Include/pyerrors.h index c6c443a2d7d0f..77d791427d492 100644 --- a/Include/pyerrors.h +++ b/Include/pyerrors.h @@ -60,11 +60,14 @@ PyAPI_FUNC(const char *) PyExceptionClass_Name(PyObject *); #define PyExceptionInstance_Class(x) ((PyObject*)Py_TYPE(x)) +#define _PyBaseExceptionGroup_Check(x) \ + PyObject_TypeCheck(x, (PyTypeObject *)PyExc_BaseExceptionGroup) /* Predefined exceptions */ PyAPI_DATA(PyObject *) PyExc_BaseException; PyAPI_DATA(PyObject *) PyExc_Exception; +PyAPI_DATA(PyObject *) PyExc_BaseExceptionGroup; #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03050000 PyAPI_DATA(PyObject *) PyExc_StopAsyncIteration; #endif diff --git a/Lib/test/exception_hierarchy.txt b/Lib/test/exception_hierarchy.txt index cf54454e71afa..5c0bfda373794 100644 --- a/Lib/test/exception_hierarchy.txt +++ b/Lib/test/exception_hierarchy.txt @@ -2,7 +2,9 @@ BaseException ??? SystemExit ??? KeyboardInterrupt ??? GeneratorExit + ??? BaseExceptionGroup ??? Exception + ??? ExceptionGroup [BaseExceptionGroup] ??? StopIteration ??? StopAsyncIteration ??? ArithmeticError diff --git a/Lib/test/test_descr.py b/Lib/test/test_descr.py index a5404b30d2459..a4131bec602ea 100644 --- a/Lib/test/test_descr.py +++ b/Lib/test/test_descr.py @@ -4032,7 +4032,11 @@ def test_builtin_bases(self): for tp in builtin_types: object.__getattribute__(tp, "__bases__") if tp is not object: - self.assertEqual(len(tp.__bases__), 1, tp) + if tp is ExceptionGroup: + num_bases = 2 + else: + num_bases = 1 + self.assertEqual(len(tp.__bases__), num_bases, tp) class L(list): pass diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 3524a0a797c41..8423cafa8c796 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -668,7 +668,7 @@ def non_Python_modules(): r""" >>> import builtins >>> tests = doctest.DocTestFinder().find(builtins) - >>> 820 < len(tests) < 840 # approximate number of objects with docstrings + >>> 825 < len(tests) < 845 # approximate number of objects with docstrings True >>> real_tests = [t for t in tests if len(t.examples) > 0] >>> len(real_tests) # objects that actually have doctests diff --git a/Lib/test/test_exception_group.py b/Lib/test/test_exception_group.py new file mode 100644 index 0000000000000..5bb6094cde742 --- /dev/null +++ b/Lib/test/test_exception_group.py @@ -0,0 +1,808 @@ +import collections.abc +import traceback +import types +import unittest + + +class TestExceptionGroupTypeHierarchy(unittest.TestCase): + def test_exception_group_types(self): + self.assertTrue(issubclass(ExceptionGroup, Exception)) + self.assertTrue(issubclass(ExceptionGroup, BaseExceptionGroup)) + self.assertTrue(issubclass(BaseExceptionGroup, BaseException)) + + def test_exception_is_not_generic_type(self): + with self.assertRaises(TypeError): + Exception[OSError] + + def test_exception_group_is_generic_type(self): + E = OSError + self.assertIsInstance(ExceptionGroup[E], types.GenericAlias) + self.assertIsInstance(BaseExceptionGroup[E], types.GenericAlias) + + +class BadConstructorArgs(unittest.TestCase): + def test_bad_EG_construction__too_many_args(self): + MSG = 'function takes exactly 2 arguments' + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup('no errors') + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup([ValueError('no msg')]) + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup('eg', [ValueError('too')], [TypeError('many')]) + + def test_bad_EG_construction__bad_message(self): + MSG = 'argument 1 must be str, not ' + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup(ValueError(12), SyntaxError('bad syntax')) + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup(None, [ValueError(12)]) + + def test_bad_EG_construction__bad_excs_sequence(self): + MSG = 'second argument \(exceptions\) must be a sequence' + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup('errors not sequence', {ValueError(42)}) + with self.assertRaisesRegex(TypeError, MSG): + ExceptionGroup("eg", None) + + MSG = 'second argument \(exceptions\) must be a non-empty sequence' + with self.assertRaisesRegex(ValueError, MSG): + ExceptionGroup("eg", []) + + def test_bad_EG_construction__nested_non_exceptions(self): + MSG = ('Item [0-9]+ of second argument \(exceptions\)' + ' is not an exception') + with self.assertRaisesRegex(ValueError, MSG): + ExceptionGroup('expect instance, not type', [OSError]); + with self.assertRaisesRegex(ValueError, MSG): + ExceptionGroup('bad error', ["not an exception"]) + + +class InstanceCreation(unittest.TestCase): + def test_EG_wraps_Exceptions__creates_EG(self): + excs = [ValueError(1), TypeError(2)] + self.assertIs( + type(ExceptionGroup("eg", excs)), + ExceptionGroup) + + def test_BEG_wraps_Exceptions__creates_EG(self): + excs = [ValueError(1), TypeError(2)] + self.assertIs( + type(BaseExceptionGroup("beg", excs)), + ExceptionGroup) + + def test_EG_wraps_BaseException__raises_TypeError(self): + MSG= "Cannot nest BaseExceptions in an ExceptionGroup" + with self.assertRaisesRegex(TypeError, MSG): + eg = ExceptionGroup("eg", [ValueError(1), KeyboardInterrupt(2)]) + + def test_BEG_wraps_BaseException__creates_BEG(self): + beg = BaseExceptionGroup("beg", [ValueError(1), KeyboardInterrupt(2)]) + self.assertIs(type(beg), BaseExceptionGroup) + + def test_EG_subclass_wraps_anything(self): + class MyEG(ExceptionGroup): + pass + + self.assertIs( + type(MyEG("eg", [ValueError(12), TypeError(42)])), + MyEG) + self.assertIs( + type(MyEG("eg", [ValueError(12), KeyboardInterrupt(42)])), + MyEG) + + def test_BEG_subclass_wraps_anything(self): + class MyBEG(BaseExceptionGroup): + pass + + self.assertIs( + type(MyBEG("eg", [ValueError(12), TypeError(42)])), + MyBEG) + self.assertIs( + type(MyBEG("eg", [ValueError(12), KeyboardInterrupt(42)])), + MyBEG) + + +def create_simple_eg(): + excs = [] + try: + try: + raise MemoryError("context and cause for ValueError(1)") + except MemoryError as e: + raise ValueError(1) from e + except ValueError as e: + excs.append(e) + + try: + try: + raise OSError("context for TypeError") + except OSError as e: + raise TypeError(int) + except TypeError as e: + excs.append(e) + + try: + try: + raise ImportError("context for ValueError(2)") + except ImportError as e: + raise ValueError(2) + except ValueError as e: + excs.append(e) + + try: + raise ExceptionGroup('simple eg', excs) + except ExceptionGroup as e: + return e + + +class ExceptionGroupFields(unittest.TestCase): + def test_basics_ExceptionGroup_fields(self): + eg = create_simple_eg() + + # check msg + self.assertEqual(eg.message, 'simple eg') + self.assertEqual(eg.args[0], 'simple eg') + + # check cause and context + self.assertIsInstance(eg.exceptions[0], ValueError) + self.assertIsInstance(eg.exceptions[0].__cause__, MemoryError) + self.assertIsInstance(eg.exceptions[0].__context__, MemoryError) + self.assertIsInstance(eg.exceptions[1], TypeError) + self.assertIsNone(eg.exceptions[1].__cause__) + self.assertIsInstance(eg.exceptions[1].__context__, OSError) + self.assertIsInstance(eg.exceptions[2], ValueError) + self.assertIsNone(eg.exceptions[2].__cause__) + self.assertIsInstance(eg.exceptions[2].__context__, ImportError) + + # check tracebacks + line0 = create_simple_eg.__code__.co_firstlineno + tb_linenos = [line0 + 27, + [line0 + 6, line0 + 14, line0 + 22]] + self.assertEqual(eg.__traceback__.tb_lineno, tb_linenos[0]) + self.assertIsNone(eg.__traceback__.tb_next) + for i in range(3): + tb = eg.exceptions[i].__traceback__ + self.assertIsNone(tb.tb_next) + self.assertEqual(tb.tb_lineno, tb_linenos[1][i]) + + def test_fields_are_readonly(self): + eg = ExceptionGroup('eg', [TypeError(1), OSError(2)]) + + self.assertEqual(type(eg.exceptions), tuple) + + eg.message + with self.assertRaises(AttributeError): + eg.message = "new msg" + + eg.exceptions + with self.assertRaises(AttributeError): + eg.exceptions = [OSError('xyz')] + + +class ExceptionGroupTestBase(unittest.TestCase): + def assertMatchesTemplate(self, exc, exc_type, template): + """ Assert that the exception matches the template + + A template describes the shape of exc. If exc is a + leaf exception (i.e., not an exception group) then + template is an exception instance that has the + expected type and args value of exc. If exc is an + exception group, then template is a list of the + templates of its nested exceptions. + """ + if exc_type is not None: + self.assertIs(type(exc), exc_type) + + if isinstance(exc, BaseExceptionGroup): + self.assertIsInstance(template, collections.abc.Sequence) + self.assertEqual(len(exc.exceptions), len(template)) + for e, t in zip(exc.exceptions, template): + self.assertMatchesTemplate(e, None, t) + else: + self.assertIsInstance(template, BaseException) + self.assertEqual(type(exc), type(template)) + self.assertEqual(exc.args, template.args) + + +class ExceptionGroupSubgroupTests(ExceptionGroupTestBase): + def setUp(self): + self.eg = create_simple_eg() + self.eg_template = [ValueError(1), TypeError(int), ValueError(2)] + + def test_basics_subgroup_split__bad_arg_type(self): + bad_args = ["bad arg", + OSError('instance not type'), + [OSError('instance not type')],] + for arg in bad_args: + with self.assertRaises(TypeError): + self.eg.subgroup(arg) + with self.assertRaises(TypeError): + self.eg.split(arg) + + def test_basics_subgroup_by_type__passthrough(self): + eg = self.eg + self.assertIs(eg, eg.subgroup(BaseException)) + self.assertIs(eg, eg.subgroup(Exception)) + self.assertIs(eg, eg.subgroup(BaseExceptionGroup)) + self.assertIs(eg, eg.subgroup(ExceptionGroup)) + + def test_basics_subgroup_by_type__no_match(self): + self.assertIsNone(self.eg.subgroup(OSError)) + + def test_basics_subgroup_by_type__match(self): + eg = self.eg + testcases = [ + # (match_type, result_template) + (ValueError, [ValueError(1), ValueError(2)]), + (TypeError, [TypeError(int)]), + ((ValueError, TypeError), self.eg_template)] + + for match_type, template in testcases: + with self.subTest(match=match_type): + subeg = eg.subgroup(match_type) + self.assertEqual(subeg.message, eg.message) + self.assertMatchesTemplate(subeg, ExceptionGroup, template) + + def test_basics_subgroup_by_predicate__passthrough(self): + self.assertIs(self.eg, self.eg.subgroup(lambda e: True)) + + def test_basics_subgroup_by_predicate__no_match(self): + self.assertIsNone(self.eg.subgroup(lambda e: False)) + + def test_basics_subgroup_by_predicate__match(self): + eg = self.eg + testcases = [ + # (match_type, result_template) + (ValueError, [ValueError(1), ValueError(2)]), + (TypeError, [TypeError(int)]), + ((ValueError, TypeError), self.eg_template)] + + for match_type, template in testcases: + subeg = eg.subgroup(lambda e: isinstance(e, match_type)) + self.assertEqual(subeg.message, eg.message) + self.assertMatchesTemplate(subeg, ExceptionGroup, template) + + +class ExceptionGroupSplitTests(ExceptionGroupTestBase): + def setUp(self): + self.eg = create_simple_eg() + self.eg_template = [ValueError(1), TypeError(int), ValueError(2)] + + def test_basics_split_by_type__passthrough(self): + for E in [BaseException, Exception, + BaseExceptionGroup, ExceptionGroup]: + match, rest = self.eg.split(E) + self.assertMatchesTemplate( + match, ExceptionGroup, self.eg_template) + self.assertIsNone(rest) + + def test_basics_split_by_type__no_match(self): + match, rest = self.eg.split(OSError) + self.assertIsNone(match) + self.assertMatchesTemplate( + rest, ExceptionGroup, self.eg_template) + + def test_basics_split_by_type__match(self): + eg = self.eg + VE = ValueError + TE = TypeError + testcases = [ + # (matcher, match_template, rest_template) + (VE, [VE(1), VE(2)], [TE(int)]), + (TE, [TE(int)], [VE(1), VE(2)]), + ((VE, TE), self.eg_template, None), + ((OSError, VE), [VE(1), VE(2)], [TE(int)]), + ] + + for match_type, match_template, rest_template in testcases: + match, rest = eg.split(match_type) + self.assertEqual(match.message, eg.message) + self.assertMatchesTemplate( + match, ExceptionGroup, match_template) + if rest_template is not None: + self.assertEqual(rest.message, eg.message) + self.assertMatchesTemplate( + rest, ExceptionGroup, rest_template) + else: + self.assertIsNone(rest) + + def test_basics_split_by_predicate__passthrough(self): + match, rest = self.eg.split(lambda e: True) + self.assertMatchesTemplate(match, ExceptionGroup, self.eg_template) + self.assertIsNone(rest) + + def test_basics_split_by_predicate__no_match(self): + match, rest = self.eg.split(lambda e: False) + self.assertIsNone(match) + self.assertMatchesTemplate(rest, ExceptionGroup, self.eg_template) + + def test_basics_split_by_predicate__match(self): + eg = self.eg + VE = ValueError + TE = TypeError + testcases = [ + # (matcher, match_template, rest_template) + (VE, [VE(1), VE(2)], [TE(int)]), + (TE, [TE(int)], [VE(1), VE(2)]), + ((VE, TE), self.eg_template, None), + ] + + for match_type, match_template, rest_template in testcases: + match, rest = eg.split(lambda e: isinstance(e, match_type)) + self.assertEqual(match.message, eg.message) + self.assertMatchesTemplate( + match, ExceptionGroup, match_template) + if rest_template is not None: + self.assertEqual(rest.message, eg.message) + self.assertMatchesTemplate( + rest, ExceptionGroup, rest_template) + + +class DeepRecursionInSplitAndSubgroup(unittest.TestCase): + def make_deep_eg(self): + e = TypeError(1) + for i in range(2000): + e = ExceptionGroup('eg', [e]) + return e + + def test_deep_split(self): + e = self.make_deep_eg() + with self.assertRaises(RecursionError): + e.split(TypeError) + + def test_deep_subgroup(self): + e = self.make_deep_eg() + with self.assertRaises(RecursionError): + e.subgroup(TypeError) + + +def leaf_generator(exc, tbs=None): + if tbs is None: + tbs = [] + tbs.append(exc.__traceback__) + if isinstance(exc, BaseExceptionGroup): + for e in exc.exceptions: + yield from leaf_generator(e, tbs) + else: + # exc is a leaf exception and its traceback + # is the concatenation of the traceback + # segments in tbs + yield exc, tbs + tbs.pop() + + +class LeafGeneratorTest(unittest.TestCase): + # The leaf_generator is mentioned in PEP 654 as a suggestion + # on how to iterate over leaf nodes of an EG. Is is also + # used below as a test utility. So we test it here. + + def test_leaf_generator(self): + eg = create_simple_eg() + + self.assertSequenceEqual( + [e for e, _ in leaf_generator(eg)], + eg.exceptions) + + for e, tbs in leaf_generator(eg): + self.assertSequenceEqual( + tbs, [eg.__traceback__, e.__traceback__]) + + +def create_nested_eg(): + excs = [] + try: + try: + raise TypeError(bytes) + except TypeError as e: + raise ExceptionGroup("nested", [e]) + except ExceptionGroup as e: + excs.append(e) + + try: + try: + raise MemoryError('out of memory') + except MemoryError as e: + raise ValueError(1) from e + except ValueError as e: + excs.append(e) + + try: + raise ExceptionGroup("root", excs) + except ExceptionGroup as eg: + return eg + + +class NestedExceptionGroupBasicsTest(ExceptionGroupTestBase): + def test_nested_group_matches_template(self): + eg = create_nested_eg() + self.assertMatchesTemplate( + eg, + ExceptionGroup, + [[TypeError(bytes)], ValueError(1)]) + + def test_nested_group_chaining(self): + eg = create_nested_eg() + self.assertIsInstance(eg.exceptions[1].__context__, MemoryError) + self.assertIsInstance(eg.exceptions[1].__cause__, MemoryError) + self.assertIsInstance(eg.exceptions[0].__context__, TypeError) + + def test_nested_exception_group_tracebacks(self): + eg = create_nested_eg() + + line0 = create_nested_eg.__code__.co_firstlineno + for (tb, expected) in [ + (eg.__traceback__, line0 + 19), + (eg.exceptions[0].__traceback__, line0 + 6), + (eg.exceptions[1].__traceback__, line0 + 14), + (eg.exceptions[0].exceptions[0].__traceback__, line0 + 4), + ]: + self.assertEqual(tb.tb_lineno, expected) + self.assertIsNone(tb.tb_next) + + def test_iteration_full_tracebacks(self): + eg = create_nested_eg() + # check that iteration over leaves + # produces the expected tracebacks + self.assertEqual(len(list(leaf_generator(eg))), 2) + + line0 = create_nested_eg.__code__.co_firstlineno + expected_tbs = [ [line0 + 19, line0 + 6, line0 + 4], + [line0 + 19, line0 + 14]] + + for (i, (_, tbs)) in enumerate(leaf_generator(eg)): + self.assertSequenceEqual( + [tb.tb_lineno for tb in tbs], + expected_tbs[i]) + + +class ExceptionGroupSplitTestBase(ExceptionGroupTestBase): + + def split_exception_group(self, eg, types): + """ Split an EG and do some sanity checks on the result """ + self.assertIsInstance(eg, BaseExceptionGroup) + + match, rest = eg.split(types) + sg = eg.subgroup(types) + + if match is not None: + self.assertIsInstance(match, BaseExceptionGroup) + for e,_ in leaf_generator(match): + self.assertIsInstance(e, types) + + self.assertIsNotNone(sg) + self.assertIsInstance(sg, BaseExceptionGroup) + for e,_ in leaf_generator(sg): + self.assertIsInstance(e, types) + + if rest is not None: + self.assertIsInstance(rest, BaseExceptionGroup) + + def leaves(exc): + return [] if exc is None else [e for e,_ in leaf_generator(exc)] + + # match and subgroup have the same leaves + self.assertSequenceEqual(leaves(match), leaves(sg)) + + match_leaves = leaves(match) + rest_leaves = leaves(rest) + # each leaf exception of eg is in exactly one of match and rest + self.assertEqual( + len(leaves(eg)), + len(leaves(match)) + len(leaves(rest))) + + for e in leaves(eg): + self.assertNotEqual( + match and e in match_leaves, + rest and e in rest_leaves) + + # message, cause and context equal to eg + for part in [match, rest, sg]: + if part is not None: + self.assertEqual(eg.message, part.message) + self.assertIs(eg.__cause__, part.__cause__) + self.assertIs(eg.__context__, part.__context__) + self.assertIs(eg.__traceback__, part.__traceback__) + + def tbs_for_leaf(leaf, eg): + for e, tbs in leaf_generator(eg): + if e is leaf: + return tbs + + def tb_linenos(tbs): + return [tb.tb_lineno for tb in tbs if tb] + + # full tracebacks match + for part in [match, rest, sg]: + for e in leaves(part): + self.assertSequenceEqual( + tb_linenos(tbs_for_leaf(e, eg)), + tb_linenos(tbs_for_leaf(e, part))) + + return match, rest + + +class NestedExceptionGroupSplitTest(ExceptionGroupSplitTestBase): + + def test_split_by_type(self): + class MyExceptionGroup(ExceptionGroup): + pass + + def raiseVE(v): + raise ValueError(v) + + def raiseTE(t): + raise TypeError(t) + + def nested_group(): + def level1(i): + excs = [] + for f, arg in [(raiseVE, i), (raiseTE, int), (raiseVE, i+1)]: + try: + f(arg) + except Exception as e: + excs.append(e) + raise ExceptionGroup('msg1', excs) + + def level2(i): + excs = [] + for f, arg in [(level1, i), (level1, i+1), (raiseVE, i+2)]: + try: + f(arg) + except Exception as e: + excs.append(e) + raise MyExceptionGroup('msg2', excs) + + def level3(i): + excs = [] + for f, arg in [(level2, i+1), (raiseVE, i+2)]: + try: + f(arg) + except Exception as e: + excs.append(e) + raise ExceptionGroup('msg3', excs) + + level3(5) + + try: + nested_group() + except ExceptionGroup as e: + eg = e + + eg_template = [ + [ + [ValueError(6), TypeError(int), ValueError(7)], + [ValueError(7), TypeError(int), ValueError(8)], + ValueError(8), + ], + ValueError(7)] + + valueErrors_template = [ + [ + [ValueError(6), ValueError(7)], + [ValueError(7), ValueError(8)], + ValueError(8), + ], + ValueError(7)] + + typeErrors_template = [[[TypeError(int)], [TypeError(int)]]] + + self.assertMatchesTemplate(eg, ExceptionGroup, eg_template) + + # Match Nothing + match, rest = self.split_exception_group(eg, SyntaxError) + self.assertIsNone(match) + self.assertMatchesTemplate(rest, ExceptionGroup, eg_template) + + # Match Everything + match, rest = self.split_exception_group(eg, BaseException) + self.assertMatchesTemplate(match, ExceptionGroup, eg_template) + self.assertIsNone(rest) + match, rest = self.split_exception_group(eg, (ValueError, TypeError)) + self.assertMatchesTemplate(match, ExceptionGroup, eg_template) + self.assertIsNone(rest) + + # Match ValueErrors + match, rest = self.split_exception_group(eg, ValueError) + self.assertMatchesTemplate(match, ExceptionGroup, valueErrors_template) + self.assertMatchesTemplate(rest, ExceptionGroup, typeErrors_template) + + # Match TypeErrors + match, rest = self.split_exception_group(eg, (TypeError, SyntaxError)) + self.assertMatchesTemplate(match, ExceptionGroup, typeErrors_template) + self.assertMatchesTemplate(rest, ExceptionGroup, valueErrors_template) + + # Match ExceptionGroup + match, rest = eg.split(ExceptionGroup) + self.assertIs(match, eg) + self.assertIsNone(rest) + + # Match MyExceptionGroup (ExceptionGroup subclass) + match, rest = eg.split(MyExceptionGroup) + self.assertMatchesTemplate(match, ExceptionGroup, [eg_template[0]]) + self.assertMatchesTemplate(rest, ExceptionGroup, [eg_template[1]]) + + def test_split_BaseExceptionGroup(self): + def exc(ex): + try: + raise ex + except BaseException as e: + return e + + try: + raise BaseExceptionGroup( + "beg", [exc(ValueError(1)), exc(KeyboardInterrupt(2))]) + except BaseExceptionGroup as e: + beg = e + + # Match Nothing + match, rest = self.split_exception_group(beg, TypeError) + self.assertIsNone(match) + self.assertMatchesTemplate( + rest, BaseExceptionGroup, [ValueError(1), KeyboardInterrupt(2)]) + + # Match Everything + match, rest = self.split_exception_group( + beg, (ValueError, KeyboardInterrupt)) + self.assertMatchesTemplate( + match, BaseExceptionGroup, [ValueError(1), KeyboardInterrupt(2)]) + self.assertIsNone(rest) + + # Match ValueErrors + match, rest = self.split_exception_group(beg, ValueError) + self.assertMatchesTemplate( + match, ExceptionGroup, [ValueError(1)]) + self.assertMatchesTemplate( + rest, BaseExceptionGroup, [KeyboardInterrupt(2)]) + + # Match KeyboardInterrupts + match, rest = self.split_exception_group(beg, KeyboardInterrupt) + self.assertMatchesTemplate( + match, BaseExceptionGroup, [KeyboardInterrupt(2)]) + self.assertMatchesTemplate( + rest, ExceptionGroup, [ValueError(1)]) + + +class NestedExceptionGroupSubclassSplitTest(ExceptionGroupSplitTestBase): + + def test_split_ExceptionGroup_subclass_no_derive_no_new_override(self): + class EG(ExceptionGroup): + pass + + try: + try: + try: + raise TypeError(2) + except TypeError as te: + raise EG("nested", [te]) + except EG as nested: + try: + raise ValueError(1) + except ValueError as ve: + raise EG("eg", [ve, nested]) + except EG as e: + eg = e + + self.assertMatchesTemplate(eg, EG, [ValueError(1), [TypeError(2)]]) + + # Match Nothing + match, rest = self.split_exception_group(eg, OSError) + self.assertIsNone(match) + self.assertMatchesTemplate( + rest, ExceptionGroup, [ValueError(1), [TypeError(2)]]) + + # Match Everything + match, rest = self.split_exception_group(eg, (ValueError, TypeError)) + self.assertMatchesTemplate( + match, ExceptionGroup, [ValueError(1), [TypeError(2)]]) + self.assertIsNone(rest) + + # Match ValueErrors + match, rest = self.split_exception_group(eg, ValueError) + self.assertMatchesTemplate(match, ExceptionGroup, [ValueError(1)]) + self.assertMatchesTemplate(rest, ExceptionGroup, [[TypeError(2)]]) + + # Match TypeErrors + match, rest = self.split_exception_group(eg, TypeError) + self.assertMatchesTemplate(match, ExceptionGroup, [[TypeError(2)]]) + self.assertMatchesTemplate(rest, ExceptionGroup, [ValueError(1)]) + + def test_split_BaseExceptionGroup_subclass_no_derive_new_override(self): + class EG(BaseExceptionGroup): + def __new__(cls, message, excs, unused): + # The "unused" arg is here to show that split() doesn't call + # the actual class constructor from the default derive() + # implementation (it would fail on unused arg if so because + # it assumes the BaseExceptionGroup.__new__ signature). + return super().__new__(cls, message, excs) + + try: + raise EG("eg", [ValueError(1), KeyboardInterrupt(2)], "unused") + except EG as e: + eg = e + + self.assertMatchesTemplate( + eg, EG, [ValueError(1), KeyboardInterrupt(2)]) + + # Match Nothing + match, rest = self.split_exception_group(eg, OSError) + self.assertIsNone(match) + self.assertMatchesTemplate( + rest, BaseExceptionGroup, [ValueError(1), KeyboardInterrupt(2)]) + + # Match Everything + match, rest = self.split_exception_group( + eg, (ValueError, KeyboardInterrupt)) + self.assertMatchesTemplate( + match, BaseExceptionGroup, [ValueError(1), KeyboardInterrupt(2)]) + self.assertIsNone(rest) + + # Match ValueErrors + match, rest = self.split_exception_group(eg, ValueError) + self.assertMatchesTemplate(match, ExceptionGroup, [ValueError(1)]) + self.assertMatchesTemplate( + rest, BaseExceptionGroup, [KeyboardInterrupt(2)]) + + # Match KeyboardInterrupt + match, rest = self.split_exception_group(eg, KeyboardInterrupt) + self.assertMatchesTemplate( + match, BaseExceptionGroup, [KeyboardInterrupt(2)]) + self.assertMatchesTemplate(rest, ExceptionGroup, [ValueError(1)]) + + def test_split_ExceptionGroup_subclass_derive_and_new_overrides(self): + class EG(ExceptionGroup): + def __new__(cls, message, excs, code): + obj = super().__new__(cls, message, excs) + obj.code = code + return obj + + def derive(self, excs): + return EG(self.message, excs, self.code) + + try: + try: + try: + raise TypeError(2) + except TypeError as te: + raise EG("nested", [te], 101) + except EG as nested: + try: + raise ValueError(1) + except ValueError as ve: + raise EG("eg", [ve, nested], 42) + except EG as e: + eg = e + + self.assertMatchesTemplate(eg, EG, [ValueError(1), [TypeError(2)]]) + + # Match Nothing + match, rest = self.split_exception_group(eg, OSError) + self.assertIsNone(match) + self.assertMatchesTemplate(rest, EG, [ValueError(1), [TypeError(2)]]) + self.assertEqual(rest.code, 42) + self.assertEqual(rest.exceptions[1].code, 101) + + # Match Everything + match, rest = self.split_exception_group(eg, (ValueError, TypeError)) + self.assertMatchesTemplate(match, EG, [ValueError(1), [TypeError(2)]]) + self.assertEqual(match.code, 42) + self.assertEqual(match.exceptions[1].code, 101) + self.assertIsNone(rest) + + # Match ValueErrors + match, rest = self.split_exception_group(eg, ValueError) + self.assertMatchesTemplate(match, EG, [ValueError(1)]) + self.assertEqual(match.code, 42) + self.assertMatchesTemplate(rest, EG, [[TypeError(2)]]) + self.assertEqual(rest.code, 42) + self.assertEqual(rest.exceptions[0].code, 101) + + # Match TypeErrors + match, rest = self.split_exception_group(eg, TypeError) + self.assertMatchesTemplate(match, EG, [[TypeError(2)]]) + self.assertEqual(match.code, 42) + self.assertEqual(match.exceptions[0].code, 101) + self.assertMatchesTemplate(rest, EG, [ValueError(1)]) + self.assertEqual(rest.code, 42) + + +if __name__ == '__main__': + unittest.main() diff --git a/Lib/test/test_pickle.py b/Lib/test/test_pickle.py index 8775ff4b79157..057af21e71fe4 100644 --- a/Lib/test/test_pickle.py +++ b/Lib/test/test_pickle.py @@ -489,7 +489,9 @@ def test_exceptions(self): ResourceWarning, StopAsyncIteration, RecursionError, - EncodingWarning): + EncodingWarning, + BaseExceptionGroup, + ExceptionGroup): continue if exc is not OSError and issubclass(exc, OSError): self.assertEqual(reverse_mapping('builtins', name), diff --git a/Lib/test/test_stable_abi_ctypes.py b/Lib/test/test_stable_abi_ctypes.py index 750aa18108327..1e27bcaf889a2 100644 --- a/Lib/test/test_stable_abi_ctypes.py +++ b/Lib/test/test_stable_abi_ctypes.py @@ -198,6 +198,7 @@ def test_available_symbols(self): "PyExc_AssertionError", "PyExc_AttributeError", "PyExc_BaseException", + "PyExc_BaseExceptionGroup", "PyExc_BlockingIOError", "PyExc_BrokenPipeError", "PyExc_BufferError", diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-09-26-18-18-50.bpo-45292.aX5HVr.rst b/Misc/NEWS.d/next/Core and Builtins/2021-09-26-18-18-50.bpo-45292.aX5HVr.rst new file mode 100644 index 0000000000000..ee48b6d5105c5 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-09-26-18-18-50.bpo-45292.aX5HVr.rst @@ -0,0 +1 @@ +Implement :pep:`654` Add :class:`ExceptionGroup` and :class:`BaseExceptionGroup`. diff --git a/Misc/stable_abi.txt b/Misc/stable_abi.txt index 23e5b96a0e8a7..9f5a85bdec40f 100644 --- a/Misc/stable_abi.txt +++ b/Misc/stable_abi.txt @@ -619,6 +619,8 @@ data PyExc_AttributeError added 3.2 data PyExc_BaseException added 3.2 +data PyExc_BaseExceptionGroup + added 3.11 data PyExc_BufferError added 3.2 data PyExc_BytesWarning diff --git a/Objects/exceptions.c b/Objects/exceptions.c index a9ea42c98422d..a5459da89a073 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -6,6 +6,7 @@ #define PY_SSIZE_T_CLEAN #include +#include #include "pycore_initconfig.h" #include "pycore_object.h" #include "structmember.h" // PyMemberDef @@ -540,7 +541,7 @@ StopIteration_clear(PyStopIterationObject *self) static void StopIteration_dealloc(PyStopIterationObject *self) { - _PyObject_GC_UNTRACK(self); + PyObject_GC_UnTrack(self); StopIteration_clear(self); Py_TYPE(self)->tp_free((PyObject *)self); } @@ -629,6 +630,516 @@ ComplexExtendsException(PyExc_BaseException, SystemExit, SystemExit, 0, 0, SystemExit_members, 0, 0, "Request to exit from the interpreter."); +/* + * BaseExceptionGroup extends BaseException + * ExceptionGroup extends BaseExceptionGroup and Exception + */ + + +static inline PyBaseExceptionGroupObject* +_PyBaseExceptionGroupObject_cast(PyObject *exc) +{ + assert(_PyBaseExceptionGroup_Check(exc)); + return (PyBaseExceptionGroupObject *)exc; +} + +static PyObject * +BaseExceptionGroup_new(PyTypeObject *type, PyObject *args, PyObject *kwds) +{ + struct _Py_exc_state *state = get_exc_state(); + PyTypeObject *PyExc_ExceptionGroup = + (PyTypeObject*)state->PyExc_ExceptionGroup; + + PyObject *message = NULL; + PyObject *exceptions = NULL; + + if (!PyArg_ParseTuple(args, "UO", &message, &exceptions)) { + return NULL; + } + + if (!PySequence_Check(exceptions)) { + PyErr_SetString( + PyExc_TypeError, + "second argument (exceptions) must be a sequence"); + return NULL; + } + + exceptions = PySequence_Tuple(exceptions); + if (!exceptions) { + return NULL; + } + + /* We are now holding a ref to the exceptions tuple */ + + Py_ssize_t numexcs = PyTuple_GET_SIZE(exceptions); + if (numexcs == 0) { + PyErr_SetString( + PyExc_ValueError, + "second argument (exceptions) must be a non-empty sequence"); + goto error; + } + + bool nested_base_exceptions = false; + for (Py_ssize_t i = 0; i < numexcs; i++) { + PyObject *exc = PyTuple_GET_ITEM(exceptions, i); + if (!exc) { + goto error; + } + if (!PyExceptionInstance_Check(exc)) { + PyErr_Format( + PyExc_ValueError, + "Item %d of second argument (exceptions) is not an exception", + i); + goto error; + } + int is_nonbase_exception = PyObject_IsInstance(exc, PyExc_Exception); + if (is_nonbase_exception < 0) { + goto error; + } + else if (is_nonbase_exception == 0) { + nested_base_exceptions = true; + } + } + + PyTypeObject *cls = type; + if (cls == PyExc_ExceptionGroup) { + if (nested_base_exceptions) { + PyErr_SetString(PyExc_TypeError, + "Cannot nest BaseExceptions in an ExceptionGroup"); + goto error; + } + } + else if (cls == (PyTypeObject*)PyExc_BaseExceptionGroup) { + if (!nested_base_exceptions) { + /* All nested exceptions are Exception subclasses, + * wrap them in an ExceptionGroup + */ + cls = PyExc_ExceptionGroup; + } + } + else { + /* Do nothing - we don't interfere with subclasses */ + } + + if (!cls) { + /* Don't crash during interpreter shutdown + * (PyExc_ExceptionGroup may have been cleared) + */ + cls = (PyTypeObject*)PyExc_BaseExceptionGroup; + } + PyBaseExceptionGroupObject *self = + _PyBaseExceptionGroupObject_cast(BaseException_new(cls, args, kwds)); + if (!self) { + goto error; + } + + self->msg = Py_NewRef(message); + self->excs = exceptions; + return (PyObject*)self; +error: + Py_DECREF(exceptions); + return NULL; +} + +static int +BaseExceptionGroup_init(PyBaseExceptionGroupObject *self, + PyObject *args, PyObject *kwds) +{ + if (!_PyArg_NoKeywords(Py_TYPE(self)->tp_name, kwds)) { + return -1; + } + if (BaseException_init((PyBaseExceptionObject *)self, args, kwds) == -1) { + return -1; + } + return 0; +} + +static int +BaseExceptionGroup_clear(PyBaseExceptionGroupObject *self) +{ + Py_CLEAR(self->msg); + Py_CLEAR(self->excs); + return BaseException_clear((PyBaseExceptionObject *)self); +} + +static void +BaseExceptionGroup_dealloc(PyBaseExceptionGroupObject *self) +{ + _PyObject_GC_UNTRACK(self); + BaseExceptionGroup_clear(self); + Py_TYPE(self)->tp_free((PyObject *)self); +} + +static int +BaseExceptionGroup_traverse(PyBaseExceptionGroupObject *self, + visitproc visit, void *arg) +{ + Py_VISIT(self->msg); + Py_VISIT(self->excs); + return BaseException_traverse((PyBaseExceptionObject *)self, visit, arg); +} + +static PyObject * +BaseExceptionGroup_str(PyBaseExceptionGroupObject *self) +{ + assert(self->msg); + assert(PyUnicode_Check(self->msg)); + return Py_NewRef(self->msg); +} + +static PyObject * +BaseExceptionGroup_derive(PyObject *self_, PyObject *args) +{ + PyBaseExceptionGroupObject *self = _PyBaseExceptionGroupObject_cast(self_); + PyObject *excs = NULL; + if (!PyArg_ParseTuple(args, "O", &excs)) { + return NULL; + } + PyObject *init_args = PyTuple_Pack(2, self->msg, excs); + if (!init_args) { + return NULL; + } + PyObject *eg = PyObject_CallObject( + PyExc_BaseExceptionGroup, init_args); + Py_DECREF(init_args); + return eg; +} + +static int +exceptiongroup_subset( + PyBaseExceptionGroupObject *_orig, PyObject *excs, PyObject **result) +{ + /* Sets *result to an ExceptionGroup wrapping excs with metadata from + * _orig. If excs is empty, sets *result to NULL. + * Returns 0 on success and -1 on error. + + * This function is used by split() to construct the match/rest parts, + * so excs is the matching or non-matching sub-sequence of orig->excs + * (this function does not verify that it is a subsequence). + */ + PyObject *orig = (PyObject *)_orig; + + *result = NULL; + Py_ssize_t num_excs = PySequence_Size(excs); + if (num_excs < 0) { + return -1; + } + else if (num_excs == 0) { + return 0; + } + + PyObject *eg = PyObject_CallMethod( + orig, "derive", "(O)", excs); + if (!eg) { + return -1; + } + + if (!_PyBaseExceptionGroup_Check(eg)) { + PyErr_SetString(PyExc_TypeError, + "derive must return an instance of BaseExceptionGroup"); + goto error; + } + + /* Now we hold a reference to the new eg */ + + PyObject *tb = PyException_GetTraceback(orig); + if (tb) { + int res = PyException_SetTraceback(eg, tb); + Py_DECREF(tb); + if (res == -1) { + goto error; + } + } + PyException_SetContext(eg, PyException_GetContext(orig)); + PyException_SetCause(eg, PyException_GetCause(orig)); + *result = eg; + return 0; +error: + Py_DECREF(eg); + return -1; +} + +typedef enum { + /* Exception type or tuple of thereof */ + EXCEPTION_GROUP_MATCH_BY_TYPE = 0, + /* A PyFunction returning True for matching exceptions */ + EXCEPTION_GROUP_MATCH_BY_PREDICATE = 1, + /* An instance or container thereof, checked with equality + * This matcher type is only used internally by the + * interpreter, it is not exposed to python code */ + EXCEPTION_GROUP_MATCH_INSTANCES = 2 +} _exceptiongroup_split_matcher_type; + +static int +get_matcher_type(PyObject *value, + _exceptiongroup_split_matcher_type *type) +{ + /* the python API supports only BY_TYPE and BY_PREDICATE */ + if (PyExceptionClass_Check(value) || + PyTuple_CheckExact(value)) { + *type = EXCEPTION_GROUP_MATCH_BY_TYPE; + return 0; + } + if (PyFunction_Check(value)) { + *type = EXCEPTION_GROUP_MATCH_BY_PREDICATE; + return 0; + } + PyErr_SetString( + PyExc_TypeError, + "expected a function, exception type or tuple of exception types"); + return -1; +} + +static int +exceptiongroup_split_check_match(PyObject *exc, + _exceptiongroup_split_matcher_type matcher_type, + PyObject *matcher_value) +{ + switch (matcher_type) { + case EXCEPTION_GROUP_MATCH_BY_TYPE: { + assert(PyExceptionClass_Check(matcher_value) || + PyTuple_CheckExact(matcher_value)); + return PyErr_GivenExceptionMatches(exc, matcher_value); + } + case EXCEPTION_GROUP_MATCH_BY_PREDICATE: { + assert(PyFunction_Check(matcher_value)); + PyObject *exc_matches = PyObject_CallOneArg(matcher_value, exc); + if (exc_matches == NULL) { + return -1; + } + int is_true = PyObject_IsTrue(exc_matches); + Py_DECREF(exc_matches); + return is_true; + } + case EXCEPTION_GROUP_MATCH_INSTANCES: { + if (PySequence_Check(matcher_value)) { + return PySequence_Contains(matcher_value, exc); + } + return matcher_value == exc; + } + } + return 0; +} + +typedef struct { + PyObject *match; + PyObject *rest; +} _exceptiongroup_split_result; + +static int +exceptiongroup_split_recursive(PyObject *exc, + _exceptiongroup_split_matcher_type matcher_type, + PyObject *matcher_value, + bool construct_rest, + _exceptiongroup_split_result *result) +{ + result->match = NULL; + result->rest = NULL; + + int is_match = exceptiongroup_split_check_match( + exc, matcher_type, matcher_value); + if (is_match < 0) { + return -1; + } + + if (is_match) { + /* Full match */ + result->match = Py_NewRef(exc); + return 0; + } + else if (!_PyBaseExceptionGroup_Check(exc)) { + /* Leaf exception and no match */ + if (construct_rest) { + result->rest = Py_NewRef(exc); + } + return 0; + } + + /* Partial match */ + + PyBaseExceptionGroupObject *eg = _PyBaseExceptionGroupObject_cast(exc); + assert(PyTuple_CheckExact(eg->excs)); + Py_ssize_t num_excs = PyTuple_Size(eg->excs); + if (num_excs < 0) { + return -1; + } + assert(num_excs > 0); /* checked in constructor, and excs is read-only */ + + int retval = -1; + PyObject *match_list = PyList_New(0); + if (!match_list) { + return -1; + } + + PyObject *rest_list = NULL; + if (construct_rest) { + rest_list = PyList_New(0); + if (!rest_list) { + goto done; + } + } + /* recursive calls */ + for (Py_ssize_t i = 0; i < num_excs; i++) { + PyObject *e = PyTuple_GET_ITEM(eg->excs, i); + _exceptiongroup_split_result rec_result; + if (Py_EnterRecursiveCall(" in exceptiongroup_split_recursive")) { + goto done; + } + if (exceptiongroup_split_recursive( + e, matcher_type, matcher_value, + construct_rest, &rec_result) == -1) { + assert(!rec_result.match); + assert(!rec_result.rest); + Py_LeaveRecursiveCall(); + goto done; + } + Py_LeaveRecursiveCall(); + if (rec_result.match) { + assert(PyList_CheckExact(match_list)); + if (PyList_Append(match_list, rec_result.match) == -1) { + Py_DECREF(rec_result.match); + goto done; + } + Py_DECREF(rec_result.match); + } + if (rec_result.rest) { + assert(construct_rest); + assert(PyList_CheckExact(rest_list)); + if (PyList_Append(rest_list, rec_result.rest) == -1) { + Py_DECREF(rec_result.rest); + goto done; + } + Py_DECREF(rec_result.rest); + } + } + + /* construct result */ + if (exceptiongroup_subset(eg, match_list, &result->match) == -1) { + goto done; + } + + if (construct_rest) { + assert(PyList_CheckExact(rest_list)); + if (exceptiongroup_subset(eg, rest_list, &result->rest) == -1) { + Py_CLEAR(result->match); + goto done; + } + } + retval = 0; +done: + Py_DECREF(match_list); + Py_XDECREF(rest_list); + if (retval == -1) { + Py_CLEAR(result->match); + Py_CLEAR(result->rest); + } + return retval; +} + +static PyObject * +BaseExceptionGroup_split(PyObject *self, PyObject *args) +{ + PyObject *matcher_value = NULL; + if (!PyArg_UnpackTuple(args, "split", 1, 1, &matcher_value)) { + return NULL; + } + + _exceptiongroup_split_matcher_type matcher_type; + if (get_matcher_type(matcher_value, &matcher_type) == -1) { + return NULL; + } + + _exceptiongroup_split_result split_result; + bool construct_rest = true; + if (exceptiongroup_split_recursive( + self, matcher_type, matcher_value, + construct_rest, &split_result) == -1) { + return NULL; + } + + PyObject *result = PyTuple_Pack( + 2, + split_result.match ? split_result.match : Py_None, + split_result.rest ? split_result.rest : Py_None); + + Py_XDECREF(split_result.match); + Py_XDECREF(split_result.rest); + return result; +} + +static PyObject * +BaseExceptionGroup_subgroup(PyObject *self, PyObject *args) +{ + PyObject *matcher_value = NULL; + if (!PyArg_UnpackTuple(args, "subgroup", 1, 1, &matcher_value)) { + return NULL; + } + + _exceptiongroup_split_matcher_type matcher_type; + if (get_matcher_type(matcher_value, &matcher_type) == -1) { + return NULL; + } + + _exceptiongroup_split_result split_result; + bool construct_rest = false; + if (exceptiongroup_split_recursive( + self, matcher_type, matcher_value, + construct_rest, &split_result) == -1) { + return NULL; + } + + PyObject *result = Py_NewRef( + split_result.match ? split_result.match : Py_None); + + Py_XDECREF(split_result.match); + assert(!split_result.rest); + return result; +} + +static PyMemberDef BaseExceptionGroup_members[] = { + {"message", T_OBJECT, offsetof(PyBaseExceptionGroupObject, msg), READONLY, + PyDoc_STR("exception message")}, + {"exceptions", T_OBJECT, offsetof(PyBaseExceptionGroupObject, excs), READONLY, + PyDoc_STR("nested exceptions")}, + {NULL} /* Sentinel */ +}; + +static PyMethodDef BaseExceptionGroup_methods[] = { + {"__class_getitem__", (PyCFunction)Py_GenericAlias, + METH_O|METH_CLASS, PyDoc_STR("See PEP 585")}, + {"derive", (PyCFunction)BaseExceptionGroup_derive, METH_VARARGS}, + {"split", (PyCFunction)BaseExceptionGroup_split, METH_VARARGS}, + {"subgroup", (PyCFunction)BaseExceptionGroup_subgroup, METH_VARARGS}, + {NULL} +}; + +ComplexExtendsException(PyExc_BaseException, BaseExceptionGroup, + BaseExceptionGroup, BaseExceptionGroup_new /* new */, + BaseExceptionGroup_methods, BaseExceptionGroup_members, + 0 /* getset */, BaseExceptionGroup_str, + "A combination of multiple unrelated exceptions."); + +/* + * ExceptionGroup extends BaseExceptionGroup, Exception + */ +static PyObject* +create_exception_group_class(void) { + struct _Py_exc_state *state = get_exc_state(); + + PyObject *bases = PyTuple_Pack( + 2, PyExc_BaseExceptionGroup, PyExc_Exception); + if (bases == NULL) { + return NULL; + } + + assert(!state->PyExc_ExceptionGroup); + state->PyExc_ExceptionGroup = PyErr_NewException( + "builtins.ExceptionGroup", bases, NULL); + + Py_DECREF(bases); + return state->PyExc_ExceptionGroup; +} + /* * KeyboardInterrupt extends BaseException */ @@ -2671,6 +3182,7 @@ _PyExc_Init(PyInterpreterState *interp) } while (0) PRE_INIT(BaseException); + PRE_INIT(BaseExceptionGroup); PRE_INIT(Exception); PRE_INIT(TypeError); PRE_INIT(StopAsyncIteration); @@ -2805,8 +3317,15 @@ _PyBuiltins_AddExceptions(PyObject *bltinmod) return _PyStatus_ERR("exceptions bootstrapping error."); } + PyObject *PyExc_ExceptionGroup = create_exception_group_class(); + if (!PyExc_ExceptionGroup) { + return _PyStatus_ERR("exceptions bootstrapping error."); + } + POST_INIT(BaseException); POST_INIT(Exception); + POST_INIT(BaseExceptionGroup); + POST_INIT(ExceptionGroup); POST_INIT(TypeError); POST_INIT(StopAsyncIteration); POST_INIT(StopIteration); @@ -2885,6 +3404,13 @@ _PyBuiltins_AddExceptions(PyObject *bltinmod) #undef INIT_ALIAS } +void +_PyExc_ClearExceptionGroupType(PyInterpreterState *interp) +{ + struct _Py_exc_state *state = &interp->exc_state; + Py_CLEAR(state->PyExc_ExceptionGroup); +} + void _PyExc_Fini(PyInterpreterState *interp) { diff --git a/PC/python3dll.c b/PC/python3dll.c index d9e6fd3e7ca7c..d2a87070de5cc 100755 --- a/PC/python3dll.c +++ b/PC/python3dll.c @@ -754,6 +754,7 @@ EXPORT_DATA(PyExc_ArithmeticError) EXPORT_DATA(PyExc_AssertionError) EXPORT_DATA(PyExc_AttributeError) EXPORT_DATA(PyExc_BaseException) +EXPORT_DATA(PyExc_BaseExceptionGroup) EXPORT_DATA(PyExc_BlockingIOError) EXPORT_DATA(PyExc_BrokenPipeError) EXPORT_DATA(PyExc_BufferError) diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index c5a209abae61a..9ce845ca61d21 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -1662,6 +1662,8 @@ finalize_interp_clear(PyThreadState *tstate) { int is_main_interp = _Py_IsMainInterpreter(tstate->interp); + _PyExc_ClearExceptionGroupType(tstate->interp); + /* Clear interpreter state and all thread states */ _PyInterpreterState_Clear(tstate); From webhook-mailer at python.org Fri Oct 22 19:20:12 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Fri, 22 Oct 2021 23:20:12 -0000 Subject: [Python-checkins] bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040) Message-ID: https://github.com/python/cpython/commit/17c61045c51512add61a9e75e9c7343cf4e4fb82 commit: 17c61045c51512add61a9e75e9c7343cf4e4fb82 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-22T17:20:03-06:00 summary: bpo-45506: Normalize _PyPathConfig.stdlib_dir when calculated. (#29040) The recently added PyConfig.stdlib_dir was being set with ".." entries. When __file__ was added for from modules this caused a problem on out-of-tree builds. This PR fixes that by normalizing "stdlib_dir" when it is calculated in getpath.c. https://bugs.python.org/issue45506 files: A Lib/test/test_fileutils.py M Include/internal/pycore_fileutils.h M Lib/test/test_posixpath.py M Modules/_testinternalcapi.c M Modules/getpath.c M Python/fileutils.c diff --git a/Include/internal/pycore_fileutils.h b/Include/internal/pycore_fileutils.h index ab436ae9b007a..d1caf9c237234 100644 --- a/Include/internal/pycore_fileutils.h +++ b/Include/internal/pycore_fileutils.h @@ -80,6 +80,9 @@ extern int _Py_add_relfile(wchar_t *dirname, const wchar_t *relfile, size_t bufsize); extern size_t _Py_find_basename(const wchar_t *filename); +PyAPI_FUNC(int) _Py_normalize_path(const wchar_t *path, + wchar_t *buf, const size_t buf_len); + // Macros to protect CRT calls against instant termination when passed an // invalid parameter (bpo-23524). IPH stands for Invalid Parameter Handler. diff --git a/Lib/test/test_fileutils.py b/Lib/test/test_fileutils.py new file mode 100644 index 0000000000000..45b3f3233c617 --- /dev/null +++ b/Lib/test/test_fileutils.py @@ -0,0 +1,30 @@ +# Run tests for functions in Python/fileutils.c. + +import os +import os.path +import unittest +from test.support import import_helper + +# Skip this test if the _testcapi module isn't available. +_testcapi = import_helper.import_module('_testinternalcapi') + + +class PathTests(unittest.TestCase): + + def test_capi_normalize_path(self): + if os.name == 'nt': + raise unittest.SkipTest('Windows has its own helper for this') + else: + from .test_posixpath import PosixPathTest as posixdata + tests = posixdata.NORMPATH_CASES + for filename, expected in tests: + if not os.path.isabs(filename): + continue + with self.subTest(filename): + result = _testcapi.normalize_path(filename) + self.assertEqual(result, expected, + msg=f'input: {filename!r} expected output: {expected!r}') + + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/test/test_posixpath.py b/Lib/test/test_posixpath.py index 8d398ec010354..e4d8384ef0b4b 100644 --- a/Lib/test/test_posixpath.py +++ b/Lib/test/test_posixpath.py @@ -304,25 +304,51 @@ def test_expanduser_pwd(self): for path in ('~', '~/.local', '~vstinner/'): self.assertEqual(posixpath.expanduser(path), path) + NORMPATH_CASES = [ + ("", "."), + ("/", "/"), + ("/.", "/"), + ("/./", "/"), + ("/.//.", "/"), + ("/foo", "/foo"), + ("/foo/bar", "/foo/bar"), + ("//", "//"), + ("///", "/"), + ("///foo/.//bar//", "/foo/bar"), + ("///foo/.//bar//.//..//.//baz///", "/foo/baz"), + ("///..//./foo/.//bar", "/foo/bar"), + (".", "."), + (".//.", "."), + ("..", ".."), + ("../", ".."), + ("../foo", "../foo"), + ("../../foo", "../../foo"), + ("../foo/../bar", "../bar"), + ("../../foo/../bar/./baz/boom/..", "../../bar/baz"), + ("/..", "/"), + ("/..", "/"), + ("/../", "/"), + ("/..//", "/"), + ("//..", "//"), + ("/../foo", "/foo"), + ("/../../foo", "/foo"), + ("/../foo/../", "/"), + ("/../foo/../bar", "/bar"), + ("/../../foo/../bar/./baz/boom/..", "/bar/baz"), + ("/../../foo/../bar/./baz/boom/.", "/bar/baz/boom"), + ] + def test_normpath(self): - self.assertEqual(posixpath.normpath(""), ".") - self.assertEqual(posixpath.normpath("/"), "/") - self.assertEqual(posixpath.normpath("//"), "//") - self.assertEqual(posixpath.normpath("///"), "/") - self.assertEqual(posixpath.normpath("///foo/.//bar//"), "/foo/bar") - self.assertEqual(posixpath.normpath("///foo/.//bar//.//..//.//baz"), - "/foo/baz") - self.assertEqual(posixpath.normpath("///..//./foo/.//bar"), "/foo/bar") - - self.assertEqual(posixpath.normpath(b""), b".") - self.assertEqual(posixpath.normpath(b"/"), b"/") - self.assertEqual(posixpath.normpath(b"//"), b"//") - self.assertEqual(posixpath.normpath(b"///"), b"/") - self.assertEqual(posixpath.normpath(b"///foo/.//bar//"), b"/foo/bar") - self.assertEqual(posixpath.normpath(b"///foo/.//bar//.//..//.//baz"), - b"/foo/baz") - self.assertEqual(posixpath.normpath(b"///..//./foo/.//bar"), - b"/foo/bar") + for path, expected in self.NORMPATH_CASES: + with self.subTest(path): + result = posixpath.normpath(path) + self.assertEqual(result, expected) + + path = path.encode('utf-8') + expected = expected.encode('utf-8') + with self.subTest(path, type=bytes): + result = posixpath.normpath(path) + self.assertEqual(result, expected) @skip_if_ABSTFN_contains_backslash def test_realpath_curdir(self): diff --git a/Modules/_testinternalcapi.c b/Modules/_testinternalcapi.c index 1ca06069e1119..1f205b873beaf 100644 --- a/Modules/_testinternalcapi.c +++ b/Modules/_testinternalcapi.c @@ -14,12 +14,14 @@ #include "Python.h" #include "pycore_atomic_funcs.h" // _Py_atomic_int_get() #include "pycore_bitutils.h" // _Py_bswap32() +#include "pycore_fileutils.h" // _Py_normalize_path #include "pycore_gc.h" // PyGC_Head #include "pycore_hashtable.h" // _Py_hashtable_new() #include "pycore_initconfig.h" // _Py_GetConfigsAsDict() #include "pycore_interp.h" // _PyInterpreterState_GetConfigCopy() #include "pycore_pyerrors.h" // _Py_UTF8_Edit_Cost() #include "pycore_pystate.h" // _PyThreadState_GET() +#include "osdefs.h" // MAXPATHLEN static PyObject * @@ -366,6 +368,27 @@ test_edit_cost(PyObject *self, PyObject *Py_UNUSED(args)) } +static PyObject * +normalize_path(PyObject *self, PyObject *filename) +{ + Py_ssize_t size = -1; + wchar_t *encoded = PyUnicode_AsWideCharString(filename, &size); + if (encoded == NULL) { + return NULL; + } + + wchar_t buf[MAXPATHLEN + 1]; + int res = _Py_normalize_path(encoded, buf, Py_ARRAY_LENGTH(buf)); + PyMem_Free(encoded); + if (res != 0) { + PyErr_SetString(PyExc_ValueError, "string too long"); + return NULL; + } + + return PyUnicode_FromWideChar(buf, -1); +} + + static PyMethodDef TestMethods[] = { {"get_configs", get_configs, METH_NOARGS}, {"get_recursion_depth", get_recursion_depth, METH_NOARGS}, @@ -377,6 +400,7 @@ static PyMethodDef TestMethods[] = { {"set_config", test_set_config, METH_O}, {"test_atomic_funcs", test_atomic_funcs, METH_NOARGS}, {"test_edit_cost", test_edit_cost, METH_NOARGS}, + {"normalize_path", normalize_path, METH_O, NULL}, {NULL, NULL} /* sentinel */ }; diff --git a/Modules/getpath.c b/Modules/getpath.c index 1405023b39b58..4dbd502ddcf04 100644 --- a/Modules/getpath.c +++ b/Modules/getpath.c @@ -519,6 +519,42 @@ search_for_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig, } +static PyStatus +calculate_set_stdlib_dir(PyCalculatePath *calculate, _PyPathConfig *pathconfig) +{ + // Note that, unlike calculate_set_prefix(), here we allow a negative + // prefix_found. That means the source tree Lib dir gets used. + if (!calculate->prefix_found) { + return _PyStatus_OK(); + } + PyStatus status; + wchar_t *prefix = calculate->prefix; + if (!_Py_isabs(prefix)) { + prefix = _PyMem_RawWcsdup(prefix); + if (prefix == NULL) { + return _PyStatus_NO_MEMORY(); + } + status = absolutize(&prefix); + if (_PyStatus_EXCEPTION(status)) { + return status; + } + } + wchar_t buf[MAXPATHLEN + 1]; + int res = _Py_normalize_path(prefix, buf, Py_ARRAY_LENGTH(buf)); + if (prefix != calculate->prefix) { + PyMem_RawFree(prefix); + } + if (res < 0) { + return PATHLEN_ERR(); + } + pathconfig->stdlib_dir = _PyMem_RawWcsdup(buf); + if (pathconfig->stdlib_dir == NULL) { + return _PyStatus_NO_MEMORY(); + } + return _PyStatus_OK(); +} + + static PyStatus calculate_prefix(PyCalculatePath *calculate, _PyPathConfig *pathconfig) { @@ -1494,12 +1530,10 @@ calculate_path(PyCalculatePath *calculate, _PyPathConfig *pathconfig) } if (pathconfig->stdlib_dir == NULL) { - if (calculate->prefix_found) { - /* This must be done *before* calculate_set_prefix() is called. */ - pathconfig->stdlib_dir = _PyMem_RawWcsdup(calculate->prefix); - if (pathconfig->stdlib_dir == NULL) { - return _PyStatus_NO_MEMORY(); - } + /* This must be done *before* calculate_set_prefix() is called. */ + status = calculate_set_stdlib_dir(calculate, pathconfig); + if (_PyStatus_EXCEPTION(status)) { + return status; } } diff --git a/Python/fileutils.c b/Python/fileutils.c index 3d8f3a4f16326..ac0046cdac37c 100644 --- a/Python/fileutils.c +++ b/Python/fileutils.c @@ -2181,6 +2181,101 @@ _Py_find_basename(const wchar_t *filename) } +/* Remove navigation elements such as "." and "..". + + This is mostly a C implementation of posixpath.normpath(). + Return 0 on success. Return -1 if "orig" is too big for the buffer. */ +int +_Py_normalize_path(const wchar_t *path, wchar_t *buf, const size_t buf_len) +{ + assert(path && *path != L'\0'); + assert(*path == SEP); // an absolute path + if (wcslen(path) + 1 >= buf_len) { + return -1; + } + + int dots = -1; + int check_leading = 1; + const wchar_t *buf_start = buf; + wchar_t *buf_next = buf; + // The resulting filename will never be longer than path. + for (const wchar_t *remainder = path; *remainder != L'\0'; remainder++) { + wchar_t c = *remainder; + buf_next[0] = c; + buf_next++; + if (c == SEP) { + assert(dots <= 2); + if (dots == 2) { + // Turn "/x/y/../z" into "/x/z". + buf_next -= 4; // "/../" + assert(*buf_next == SEP); + // We cap it off at the root, so "/../spam" becomes "/spam". + if (buf_next == buf_start) { + buf_next++; + } + else { + // Move to the previous SEP in the buffer. + while (*(buf_next - 1) != SEP) { + assert(buf_next != buf_start); + buf_next--; + } + } + } + else if (dots == 1) { + // Turn "/./" into "/". + buf_next -= 2; // "./" + assert(*(buf_next - 1) == SEP); + } + else if (dots == 0) { + // Turn "//" into "/". + buf_next--; + assert(*(buf_next - 1) == SEP); + if (check_leading) { + if (buf_next - 1 == buf && *(remainder + 1) != SEP) { + // Leave a leading "//" alone, unless "///...". + buf_next++; + buf_start++; + } + check_leading = 0; + } + } + dots = 0; + } + else { + check_leading = 0; + if (dots >= 0) { + if (c == L'.' && dots < 2) { + dots++; + } + else { + dots = -1; + } + } + } + } + if (dots >= 0) { + // Strip any trailing dots and trailing slash. + buf_next -= dots + 1; // "/" or "/." or "/.." + assert(*buf_next == SEP); + if (buf_next == buf_start) { + // Leave the leading slash for root. + buf_next++; + } + else { + if (dots == 2) { + // Move to the previous SEP in the buffer. + do { + assert(buf_next != buf_start); + buf_next--; + } while (*(buf_next) != SEP); + } + } + } + *buf_next = L'\0'; + return 0; +} + + /* Get the current directory. buflen is the buffer size in wide characters including the null character. Decode the path from the locale encoding. From webhook-mailer at python.org Sat Oct 23 09:35:57 2021 From: webhook-mailer at python.org (pablogsal) Date: Sat, 23 Oct 2021 13:35:57 -0000 Subject: [Python-checkins] bpo-45574: fix warning about `print_escape` being unused (GH-29172) (#29176) Message-ID: https://github.com/python/cpython/commit/cadf06eab75c887dfc753ca80ef35cd2a7871135 commit: cadf06eab75c887dfc753ca80ef35cd2a7871135 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: pablogsal date: 2021-10-23T14:35:48+01:00 summary: bpo-45574: fix warning about `print_escape` being unused (GH-29172) (#29176) It used to be like this: ?????? ?????? 2021-10-22 ? 23 07 40 Quick `grep` tells that it is just used in one place under `Py_DEBUG`: https://github.com/python/cpython/blame/f6e8b80d20159596cf641305bad3a833bedd2f4f/Parser/tokenizer.cGH-L1047-L1051 ?????? ?????? 2021-10-22 ? 23 08 09 I am not sure, but it also looks like a private thing, it should not affect other users. Automerge-Triggered-By: GH:pablogsal (cherry picked from commit 4bc5473a42c5eae0928430930b897209492e849d) Co-authored-by: Nikita Sobolev Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst M Parser/tokenizer.c diff --git a/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst b/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst new file mode 100644 index 0000000000000..b404d24473960 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-22-23-06-33.bpo-45574.svqA84.rst @@ -0,0 +1 @@ +Fix warning about ``print_escape`` being unused. diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 53c10282ce86a..cd348487e0fd5 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -994,6 +994,7 @@ tok_underflow_file(struct tok_state *tok) { return tok->done == E_OK; } +#if defined(Py_DEBUG) static void print_escape(FILE *f, const char *s, Py_ssize_t size) { @@ -1020,6 +1021,7 @@ print_escape(FILE *f, const char *s, Py_ssize_t size) } putc('"', f); } +#endif /* Get next char, updating state; error code goes into tok->done */ From webhook-mailer at python.org Sat Oct 23 11:48:04 2021 From: webhook-mailer at python.org (jaraco) Date: Sat, 23 Oct 2021 15:48:04 -0000 Subject: [Python-checkins] bpo-45516: add protocol description to the TraversableResources documentation (#29173) Message-ID: https://github.com/python/cpython/commit/8ce20bbdd6d2b1277a5e74154fcdcef2cb0fee49 commit: 8ce20bbdd6d2b1277a5e74154fcdcef2cb0fee49 branch: main author: Filipe La?ns committer: jaraco date: 2021-10-23T11:47:55-04:00 summary: bpo-45516: add protocol description to the TraversableResources documentation (#29173) Signed-off-by: Filipe La?ns files: A Misc/NEWS.d/next/Documentation/2021-10-22-21-57-42.bpo-45516.7_RMEX.rst M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index a25f5145cac83..7a656fcb091da 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -870,6 +870,11 @@ ABC hierarchy:: .. versionadded:: 3.9 + .. abstractmethod:: files() + + Returns a :class:`importlib.abc.Traversable` object for the loaded + package. + :mod:`importlib.resources` -- Resources --------------------------------------- diff --git a/Misc/NEWS.d/next/Documentation/2021-10-22-21-57-42.bpo-45516.7_RMEX.rst b/Misc/NEWS.d/next/Documentation/2021-10-22-21-57-42.bpo-45516.7_RMEX.rst new file mode 100644 index 0000000000000..6295dd04b626a --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-22-21-57-42.bpo-45516.7_RMEX.rst @@ -0,0 +1,2 @@ +Add protocol description to the :class:`importlib.abc.TraversableResources` +documentation. From webhook-mailer at python.org Sat Oct 23 12:24:08 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 23 Oct 2021 16:24:08 -0000 Subject: [Python-checkins] Expand about soft keywords in the docs for keyword.py (GH-29139) Message-ID: https://github.com/python/cpython/commit/9e05da62241877a871c7b22ef723d7028ce03c90 commit: 9e05da62241877a871c7b22ef723d7028ce03c90 branch: main author: William Andrea committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-23T09:24:02-07:00 summary: Expand about soft keywords in the docs for keyword.py (GH-29139) Add link at the top and fix the existing links to point to the "[soft keywords](https://docs.python.org/3.10/reference/lexical_analysis.html#soft-keywords)" section created in the Python 3.10 docs. Changes should be backported to 3.10 as well. files: M Doc/library/keyword.rst diff --git a/Doc/library/keyword.rst b/Doc/library/keyword.rst index 5cae79f5dc9db..c3b4699cb05af 100644 --- a/Doc/library/keyword.rst +++ b/Doc/library/keyword.rst @@ -9,7 +9,7 @@ -------------- This module allows a Python program to determine if a string is a -:ref:`keyword `. +:ref:`keyword ` or :ref:`soft keyword `. .. function:: iskeyword(s) @@ -26,14 +26,14 @@ This module allows a Python program to determine if a string is a .. function:: issoftkeyword(s) - Return ``True`` if *s* is a Python soft :ref:`keyword `. + Return ``True`` if *s* is a Python :ref:`soft keyword `. .. versionadded:: 3.9 .. data:: softkwlist - Sequence containing all the soft :ref:`keywords ` defined for the + Sequence containing all the :ref:`soft keywords ` defined for the interpreter. If any soft keywords are defined to only be active when particular :mod:`__future__` statements are in effect, these will be included as well. From webhook-mailer at python.org Sun Oct 24 09:06:36 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 24 Oct 2021 13:06:36 -0000 Subject: [Python-checkins] bpo-45566: `test_frozen_pickle` checks all `pickle` protocols (GH-29150) Message-ID: https://github.com/python/cpython/commit/07236d562e59c6650227be18fa6ffc66b18d4741 commit: 07236d562e59c6650227be18fa6ffc66b18d4741 branch: main author: Nikita Sobolev committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-24T06:06:32-07:00 summary: bpo-45566: `test_frozen_pickle` checks all `pickle` protocols (GH-29150) Refs https://github.com/python/cpython/pull/29147 Automerge-Triggered-By: GH:ericvsmith files: A Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst M Lib/test/test_dataclasses.py diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index bdcb4a2cfd1a0..bbbb8e6c6395b 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -2859,13 +2859,26 @@ class FrozenSlotsClass: foo: str bar: int + @dataclass(frozen=True) + class FrozenWithoutSlotsClass: + foo: str + bar: int + def test_frozen_pickle(self): # bpo-43999 - assert self.FrozenSlotsClass.__slots__ == ("foo", "bar") - p = pickle.dumps(self.FrozenSlotsClass("a", 1)) - assert pickle.loads(p) == self.FrozenSlotsClass("a", 1) - + self.assertEqual(self.FrozenSlotsClass.__slots__, ("foo", "bar")) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + obj = self.FrozenSlotsClass("a", 1) + p = pickle.loads(pickle.dumps(obj, protocol=proto)) + self.assertIsNot(obj, p) + self.assertEqual(obj, p) + + obj = self.FrozenWithoutSlotsClass("a", 1) + p = pickle.loads(pickle.dumps(obj, protocol=proto)) + self.assertIsNot(obj, p) + self.assertEqual(obj, p) class TestDescriptors(unittest.TestCase): def test_set_name(self): diff --git a/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst b/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst new file mode 100644 index 0000000000000..a2ecf721800d9 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst @@ -0,0 +1 @@ +Fix ``test_frozen_pickle`` in ``test_dataclasses`` to check all ``pickle`` versions. From webhook-mailer at python.org Sun Oct 24 09:29:46 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 24 Oct 2021 13:29:46 -0000 Subject: [Python-checkins] bpo-45566: `test_frozen_pickle` checks all `pickle` protocols (GH-29150) Message-ID: https://github.com/python/cpython/commit/36971fd1f490664fb62b1fab869c5637669f0967 commit: 36971fd1f490664fb62b1fab869c5637669f0967 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-24T06:29:37-07:00 summary: bpo-45566: `test_frozen_pickle` checks all `pickle` protocols (GH-29150) Refs https://github.com/python/cpython/pull/29147 Automerge-Triggered-By: GH:ericvsmith (cherry picked from commit 07236d562e59c6650227be18fa6ffc66b18d4741) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst M Lib/test/test_dataclasses.py diff --git a/Lib/test/test_dataclasses.py b/Lib/test/test_dataclasses.py index bdcb4a2cfd1a0..bbbb8e6c6395b 100644 --- a/Lib/test/test_dataclasses.py +++ b/Lib/test/test_dataclasses.py @@ -2859,13 +2859,26 @@ class FrozenSlotsClass: foo: str bar: int + @dataclass(frozen=True) + class FrozenWithoutSlotsClass: + foo: str + bar: int + def test_frozen_pickle(self): # bpo-43999 - assert self.FrozenSlotsClass.__slots__ == ("foo", "bar") - p = pickle.dumps(self.FrozenSlotsClass("a", 1)) - assert pickle.loads(p) == self.FrozenSlotsClass("a", 1) - + self.assertEqual(self.FrozenSlotsClass.__slots__, ("foo", "bar")) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + obj = self.FrozenSlotsClass("a", 1) + p = pickle.loads(pickle.dumps(obj, protocol=proto)) + self.assertIsNot(obj, p) + self.assertEqual(obj, p) + + obj = self.FrozenWithoutSlotsClass("a", 1) + p = pickle.loads(pickle.dumps(obj, protocol=proto)) + self.assertIsNot(obj, p) + self.assertEqual(obj, p) class TestDescriptors(unittest.TestCase): def test_set_name(self): diff --git a/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst b/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst new file mode 100644 index 0000000000000..a2ecf721800d9 --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-22-12-05-21.bpo-45566.2gQ3ZB.rst @@ -0,0 +1 @@ +Fix ``test_frozen_pickle`` in ``test_dataclasses`` to check all ``pickle`` versions. From webhook-mailer at python.org Sun Oct 24 23:27:34 2021 From: webhook-mailer at python.org (tim-one) Date: Mon, 25 Oct 2021 03:27:34 -0000 Subject: [Python-checkins] bpo-45530: speed listobject.c's unsafe_tuple_compare() (GH-29076) Message-ID: https://github.com/python/cpython/commit/51ed2c56a1852cd6b09c85ba81312dc9782772ce commit: 51ed2c56a1852cd6b09c85ba81312dc9782772ce branch: main author: Tim Peters committer: tim-one date: 2021-10-24T22:27:24-05:00 summary: bpo-45530: speed listobject.c's unsafe_tuple_compare() (GH-29076) Keep track of whether unsafe_tuple_compare() calls are resolved by the very first tuple elements, and adjust strategy accordingly. This can significantly cut the number of calls made to the full-blown PyObject_RichCompareBool(), and especially when duplicates are rare. Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Core and Builtins/2021-10-20-01-28-26.bpo-45530.5r7n4m.rst M Doc/reference/expressions.rst M Doc/whatsnew/3.11.rst M Objects/listobject.c diff --git a/Doc/reference/expressions.rst b/Doc/reference/expressions.rst index d70fcb34d2168..d21a44431e52a 100644 --- a/Doc/reference/expressions.rst +++ b/Doc/reference/expressions.rst @@ -1424,6 +1424,8 @@ Note that ``a op1 b op2 c`` doesn't imply any kind of comparison between *a* and *c*, so that, e.g., ``x < y > z`` is perfectly legal (though perhaps not pretty). +.. _expressions-value-comparisons: + Value comparisons ----------------- diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index a03fff8a9e10a..7b3ce9bc7feaf 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -470,6 +470,13 @@ Changes in the Python API the ``'utf-8'`` encoding. (Contributed by Srinivas Reddy Thatiparthy (?????????? ?????? ?????????) in :issue:`41137`.) +* When sorting using tuples as keys, the order of the result may differ + from earlier releases if the tuple elements don't define a total + ordering (see :ref:`expressions-value-comparisons` for + information on total ordering). It's generally true that the result + of sorting simply isn't well-defined in the absence of a total ordering + on list elements. + Build Changes ============= diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-20-01-28-26.bpo-45530.5r7n4m.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-20-01-28-26.bpo-45530.5r7n4m.rst new file mode 100644 index 0000000000000..a8b155e7ccfcd --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-20-01-28-26.bpo-45530.5r7n4m.rst @@ -0,0 +1,8 @@ +Cases of sorting using tuples as keys may now be significantly faster +in some cases. Patch by Tim Peters. + +The order of the result may differ from earlier releases if the tuple +elements don't define a total ordering (see +:ref:`expressions-value-comparisons` for information on +total ordering). It's generally true that the result of sorting simply +isn't well-defined in the absence of a total ordering on list elements. diff --git a/Objects/listobject.c b/Objects/listobject.c index e0cae87f45781..05743cde55191 100644 --- a/Objects/listobject.c +++ b/Objects/listobject.c @@ -1220,6 +1220,13 @@ struct s_MergeState { * of tuples. It may be set to safe_object_compare, but the idea is that hopefully * we can assume more, and use one of the special-case compares. */ int (*tuple_elem_compare)(PyObject *, PyObject *, MergeState *); + + /* Used by unsafe_tuple_compare to record whether the very first tuple + * elements resolved the last comparison attempt. If so, next time a + * method that may avoid PyObject_RichCompareBool() entirely is tried. + * 0 for false, 1 for true. + */ + int first_tuple_items_resolved_it; }; /* binarysort is the best method for sorting small arrays: it does @@ -2190,7 +2197,24 @@ unsafe_float_compare(PyObject *v, PyObject *w, MergeState *ms) * using the same pre-sort check as we use for ms->key_compare, * but run on the list [x[0] for x in L]. This allows us to optimize compares * on two levels (as long as [x[0] for x in L] is type-homogeneous.) The idea is - * that most tuple compares don't involve x[1:]. */ + * that most tuple compares don't involve x[1:]. + * However, that may not be right. When it is right, we can win by calling the + * relatively cheap ms->tuple_elem_compare on the first pair of elements, to + * see whether v[0] < w[0] or w[0] < v[0]. If either are so, we're done. + * Else we proceed as in the tuple compare, comparing the remaining pairs via + * the probably more expensive PyObject_RichCompareBool(..., Py_EQ) until (if + * ever) that says "no, not equal!". Then, if we're still on the first pair, + * ms->tuple_elem_compare can resolve it, else PyObject_RichCompareBool(..., + * Py_LT) finishes the job. + * In any case, ms->first_tuple_items_resolved_it keeps track of whether the + * most recent tuple comparison was resolved by the first pair. If so, the + * next attempt starts by trying the cheap tests on the first pair again, else + * PyObject_RichCompareBool(..., Py_EQ) is used from the start. + * There are cases where PyObject_RichCompareBool(..., Py_EQ) is much cheaper! + * For example, that can return "almost immediately" if passed the same + * object twice (it special-cases object identity for Py_EQ), which can, + * potentially, be unboundedly faster than ms->tuple_elem_compare. + */ static int unsafe_tuple_compare(PyObject *v, PyObject *w, MergeState *ms) { @@ -2206,26 +2230,52 @@ unsafe_tuple_compare(PyObject *v, PyObject *w, MergeState *ms) vt = (PyTupleObject *)v; wt = (PyTupleObject *)w; + i = 0; + if (ms->first_tuple_items_resolved_it) { + /* See whether fast compares of the first elements settle it. */ + k = ms->tuple_elem_compare(vt->ob_item[0], wt->ob_item[0], ms); + if (k) /* error, or v < w */ + return k; + k = ms->tuple_elem_compare(wt->ob_item[0], vt->ob_item[0], ms); + if (k > 0) /* w < v */ + return 0; + if (k < 0) /* error */ + return -1; + /* We have + * not (v[0] < w[0]) and not (w[0] < v[0]) + * which implies, for a total order, that the first elements are + * equal. So skip them in the loop. + */ + i = 1; + ms->first_tuple_items_resolved_it = 0; + } + /* Now first_tuple_items_resolved_it was 0 on entry, or was forced to 0 + * at the end of the `if` block just above. + */ + assert(! ms->first_tuple_items_resolved_it); vlen = Py_SIZE(vt); wlen = Py_SIZE(wt); - - for (i = 0; i < vlen && i < wlen; i++) { + for (; i < vlen && i < wlen; i++) { k = PyObject_RichCompareBool(vt->ob_item[i], wt->ob_item[i], Py_EQ); + if (!k) { /* not equal */ + if (i) { + return PyObject_RichCompareBool(vt->ob_item[i], wt->ob_item[i], + Py_LT); + } + else { + ms->first_tuple_items_resolved_it = 1; + return ms->tuple_elem_compare(vt->ob_item[0], wt->ob_item[0], + ms); + } + } if (k < 0) return -1; - if (!k) - break; } + /* all equal until we fell off the end */ + return vlen < wlen; - if (i >= vlen || i >= wlen) - return vlen < wlen; - - if (i == 0) - return ms->tuple_elem_compare(vt->ob_item[i], wt->ob_item[i], ms); - else - return PyObject_RichCompareBool(vt->ob_item[i], wt->ob_item[i], Py_LT); -} + } /* An adaptive, stable, natural mergesort. See listsort.txt. * Returns Py_None on success, NULL on error. Even in case of error, the @@ -2408,6 +2458,7 @@ list_sort_impl(PyListObject *self, PyObject *keyfunc, int reverse) } ms.key_compare = unsafe_tuple_compare; + ms.first_tuple_items_resolved_it = 1; /* be optimistic */ } } /* End of pre-sort check: ms is now set properly! */ From webhook-mailer at python.org Mon Oct 25 04:25:47 2021 From: webhook-mailer at python.org (miss-islington) Date: Mon, 25 Oct 2021 08:25:47 -0000 Subject: [Python-checkins] bpo-45548: Remove _math.c workarounds for pre-C99 libm (GH-29179) Message-ID: https://github.com/python/cpython/commit/fa26245a1c1aa938cce391348d6bd879da357522 commit: fa26245a1c1aa938cce391348d6bd879da357522 branch: main author: Christian Heimes committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-25T01:25:27-07:00 summary: bpo-45548: Remove _math.c workarounds for pre-C99 libm (GH-29179) The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible ``libm`` and no longer ship with workarounds for missing acosh, asinh, expm1, and log1p functions. The changeset also removes ``_math.c`` and moves the last remaining workaround into ``_math.h``. This simplifies static builds with ``Modules/Setup`` and resolves symbol conflicts. Co-authored-by: Mark Dickinson Co-authored-by: Brett Cannon Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-24-21-49-49.bpo-45548.UWx0UC.rst D Modules/_math.c M Makefile.pre.in M Modules/Setup M Modules/_math.h M Modules/cmathmodule.c M Modules/mathmodule.c M PCbuild/pythoncore.vcxproj M PCbuild/pythoncore.vcxproj.filters M configure M configure.ac M setup.py diff --git a/Makefile.pre.in b/Makefile.pre.in index f03f535f6faa6..4ee8fddbe14ee 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -611,10 +611,6 @@ pybuilddir.txt: $(BUILDPYTHON) exit 1 ; \ fi -# This is shared by the math and cmath modules -Modules/_math.o: Modules/_math.c Modules/_math.h - $(CC) -c $(CCSHARED) $(PY_CORE_CFLAGS) -o $@ $< - # blake2s is auto-generated from blake2b $(srcdir)/Modules/_blake2/blake2s_impl.c: $(srcdir)/Modules/_blake2/blake2b_impl.c $(srcdir)/Modules/_blake2/blake2b2s.py $(PYTHON_FOR_REGEN) $(srcdir)/Modules/_blake2/blake2b2s.py @@ -625,7 +621,7 @@ $(srcdir)/Modules/_blake2/blake2s_impl.c: $(srcdir)/Modules/_blake2/blake2b_impl # -s, --silent or --quiet is always the first char. # Under BSD make, MAKEFLAGS might be " -s -v x=y". # Ignore macros passed by GNU make, passed after -- -sharedmods: $(BUILDPYTHON) pybuilddir.txt Modules/_math.o +sharedmods: $(BUILDPYTHON) pybuilddir.txt @case "`echo X $$MAKEFLAGS | sed 's/^X //;s/ -- .*//'`" in \ *\ -s*|s*) quiet="-q";; \ *) quiet="";; \ diff --git a/Misc/NEWS.d/next/Build/2021-10-24-21-49-49.bpo-45548.UWx0UC.rst b/Misc/NEWS.d/next/Build/2021-10-24-21-49-49.bpo-45548.UWx0UC.rst new file mode 100644 index 0000000000000..15fd56615ffa9 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-24-21-49-49.bpo-45548.UWx0UC.rst @@ -0,0 +1,3 @@ +The :mod:`math` and :mod:`cmath` implementation now require a C99 compatible +``libm`` and no longer ship with workarounds for missing acosh, asinh, atanh, +expm1, and log1p functions. diff --git a/Modules/Setup b/Modules/Setup index d06d733fc9cbb..99c7d4e909e18 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -171,8 +171,8 @@ time timemodule.c #array arraymodule.c #audioop audioop.c #binascii binascii.c -#cmath cmathmodule.c _math.c # -lm -#math mathmodule.c _math.c # -lm +#cmath cmathmodule.c # -lm +#math mathmodule.c # -lm #pyexpat -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c #unicodedata unicodedata.c diff --git a/Modules/_math.c b/Modules/_math.c deleted file mode 100644 index c1936a1088a24..0000000000000 --- a/Modules/_math.c +++ /dev/null @@ -1,270 +0,0 @@ -/* Definitions of some C99 math library functions, for those platforms - that don't implement these functions already. */ - -#ifndef Py_BUILD_CORE_BUILTIN -# define Py_BUILD_CORE_MODULE 1 -#endif - -#include "Python.h" -#include -#include "_math.h" - -/* The following copyright notice applies to the original - implementations of acosh, asinh and atanh. */ - -/* - * ==================================================== - * Copyright (C) 1993 by Sun Microsystems, Inc. All rights reserved. - * - * Developed at SunPro, a Sun Microsystems, Inc. business. - * Permission to use, copy, modify, and distribute this - * software is freely granted, provided that this notice - * is preserved. - * ==================================================== - */ - -#if !defined(HAVE_ACOSH) || !defined(HAVE_ASINH) -static const double ln2 = 6.93147180559945286227E-01; -static const double two_pow_p28 = 268435456.0; /* 2**28 */ -#endif -#if !defined(HAVE_ASINH) || !defined(HAVE_ATANH) -static const double two_pow_m28 = 3.7252902984619141E-09; /* 2**-28 */ -#endif -#if !defined(HAVE_ATANH) && !defined(Py_NAN) -static const double zero = 0.0; -#endif - - -#ifndef HAVE_ACOSH -/* acosh(x) - * Method : - * Based on - * acosh(x) = log [ x + sqrt(x*x-1) ] - * we have - * acosh(x) := log(x)+ln2, if x is large; else - * acosh(x) := log(2x-1/(sqrt(x*x-1)+x)) if x>2; else - * acosh(x) := log1p(t+sqrt(2.0*t+t*t)); where t=x-1. - * - * Special cases: - * acosh(x) is NaN with signal if x<1. - * acosh(NaN) is NaN without signal. - */ - -double -_Py_acosh(double x) -{ - if (Py_IS_NAN(x)) { - return x+x; - } - if (x < 1.) { /* x < 1; return a signaling NaN */ - errno = EDOM; -#ifdef Py_NAN - return Py_NAN; -#else - return (x-x)/(x-x); -#endif - } - else if (x >= two_pow_p28) { /* x > 2**28 */ - if (Py_IS_INFINITY(x)) { - return x+x; - } - else { - return log(x) + ln2; /* acosh(huge)=log(2x) */ - } - } - else if (x == 1.) { - return 0.0; /* acosh(1) = 0 */ - } - else if (x > 2.) { /* 2 < x < 2**28 */ - double t = x * x; - return log(2.0 * x - 1.0 / (x + sqrt(t - 1.0))); - } - else { /* 1 < x <= 2 */ - double t = x - 1.0; - return m_log1p(t + sqrt(2.0 * t + t * t)); - } -} -#endif /* HAVE_ACOSH */ - - -#ifndef HAVE_ASINH -/* asinh(x) - * Method : - * Based on - * asinh(x) = sign(x) * log [ |x| + sqrt(x*x+1) ] - * we have - * asinh(x) := x if 1+x*x=1, - * := sign(x)*(log(x)+ln2) for large |x|, else - * := sign(x)*log(2|x|+1/(|x|+sqrt(x*x+1))) if|x|>2, else - * := sign(x)*log1p(|x| + x^2/(1 + sqrt(1+x^2))) - */ - -double -_Py_asinh(double x) -{ - double w; - double absx = fabs(x); - - if (Py_IS_NAN(x) || Py_IS_INFINITY(x)) { - return x+x; - } - if (absx < two_pow_m28) { /* |x| < 2**-28 */ - return x; /* return x inexact except 0 */ - } - if (absx > two_pow_p28) { /* |x| > 2**28 */ - w = log(absx) + ln2; - } - else if (absx > 2.0) { /* 2 < |x| < 2**28 */ - w = log(2.0 * absx + 1.0 / (sqrt(x * x + 1.0) + absx)); - } - else { /* 2**-28 <= |x| < 2= */ - double t = x*x; - w = m_log1p(absx + t / (1.0 + sqrt(1.0 + t))); - } - return copysign(w, x); - -} -#endif /* HAVE_ASINH */ - - -#ifndef HAVE_ATANH -/* atanh(x) - * Method : - * 1.Reduced x to positive by atanh(-x) = -atanh(x) - * 2.For x>=0.5 - * 1 2x x - * atanh(x) = --- * log(1 + -------) = 0.5 * log1p(2 * -------) - * 2 1 - x 1 - x - * - * For x<0.5 - * atanh(x) = 0.5*log1p(2x+2x*x/(1-x)) - * - * Special cases: - * atanh(x) is NaN if |x| >= 1 with signal; - * atanh(NaN) is that NaN with no signal; - * - */ - -double -_Py_atanh(double x) -{ - double absx; - double t; - - if (Py_IS_NAN(x)) { - return x+x; - } - absx = fabs(x); - if (absx >= 1.) { /* |x| >= 1 */ - errno = EDOM; -#ifdef Py_NAN - return Py_NAN; -#else - return x / zero; -#endif - } - if (absx < two_pow_m28) { /* |x| < 2**-28 */ - return x; - } - if (absx < 0.5) { /* |x| < 0.5 */ - t = absx+absx; - t = 0.5 * m_log1p(t + t*absx / (1.0 - absx)); - } - else { /* 0.5 <= |x| <= 1.0 */ - t = 0.5 * m_log1p((absx + absx) / (1.0 - absx)); - } - return copysign(t, x); -} -#endif /* HAVE_ATANH */ - - -#ifndef HAVE_EXPM1 -/* Mathematically, expm1(x) = exp(x) - 1. The expm1 function is designed - to avoid the significant loss of precision that arises from direct - evaluation of the expression exp(x) - 1, for x near 0. */ - -double -_Py_expm1(double x) -{ - /* For abs(x) >= log(2), it's safe to evaluate exp(x) - 1 directly; this - also works fine for infinities and nans. - - For smaller x, we can use a method due to Kahan that achieves close to - full accuracy. - */ - - if (fabs(x) < 0.7) { - double u; - u = exp(x); - if (u == 1.0) - return x; - else - return (u - 1.0) * x / log(u); - } - else - return exp(x) - 1.0; -} -#endif /* HAVE_EXPM1 */ - - -/* log1p(x) = log(1+x). The log1p function is designed to avoid the - significant loss of precision that arises from direct evaluation when x is - small. */ - -double -_Py_log1p(double x) -{ -#ifdef HAVE_LOG1P - /* Some platforms supply a log1p function but don't respect the sign of - zero: log1p(-0.0) gives 0.0 instead of the correct result of -0.0. - - To save fiddling with configure tests and platform checks, we handle the - special case of zero input directly on all platforms. - */ - if (x == 0.0) { - return x; - } - else { - return log1p(x); - } -#else - /* For x small, we use the following approach. Let y be the nearest float - to 1+x, then - - 1+x = y * (1 - (y-1-x)/y) - - so log(1+x) = log(y) + log(1-(y-1-x)/y). Since (y-1-x)/y is tiny, the - second term is well approximated by (y-1-x)/y. If abs(x) >= - DBL_EPSILON/2 or the rounding-mode is some form of round-to-nearest - then y-1-x will be exactly representable, and is computed exactly by - (y-1)-x. - - If abs(x) < DBL_EPSILON/2 and the rounding mode is not known to be - round-to-nearest then this method is slightly dangerous: 1+x could be - rounded up to 1+DBL_EPSILON instead of down to 1, and in that case - y-1-x will not be exactly representable any more and the result can be - off by many ulps. But this is easily fixed: for a floating-point - number |x| < DBL_EPSILON/2., the closest floating-point number to - log(1+x) is exactly x. - */ - - double y; - if (fabs(x) < DBL_EPSILON / 2.) { - return x; - } - else if (-0.5 <= x && x <= 1.) { - /* WARNING: it's possible that an overeager compiler - will incorrectly optimize the following two lines - to the equivalent of "return log(1.+x)". If this - happens, then results from log1p will be inaccurate - for small x. */ - y = 1.+x; - return log(y) - ((y - 1.) - x) / y; - } - else { - /* NaNs and infinities should end up here */ - return log(1.+x); - } -#endif /* ifdef HAVE_LOG1P */ -} - diff --git a/Modules/_math.h b/Modules/_math.h index 398b7e8874af0..4a6bc223ef5fb 100644 --- a/Modules/_math.h +++ b/Modules/_math.h @@ -1,41 +1,24 @@ -#ifdef HAVE_ACOSH -# define m_acosh acosh -#else -/* if the system doesn't have acosh, use the substitute - function defined in Modules/_math.c. */ -double _Py_acosh(double x); -# define m_acosh _Py_acosh -#endif +/* log1p(x) = log(1+x). The log1p function is designed to avoid the + significant loss of precision that arises from direct evaluation when x is + small. Use the substitute from _math.h on all platforms: it includes + workarounds for buggy handling of zeros. + */ -#ifdef HAVE_ASINH -# define m_asinh asinh -#else -/* if the system doesn't have asinh, use the substitute - function defined in Modules/_math.c. */ -double _Py_asinh(double x); -# define m_asinh _Py_asinh -#endif +static double +_Py_log1p(double x) +{ + /* Some platforms supply a log1p function but don't respect the sign of + zero: log1p(-0.0) gives 0.0 instead of the correct result of -0.0. -#ifdef HAVE_ATANH -# define m_atanh atanh -#else -/* if the system doesn't have atanh, use the substitute - function defined in Modules/_math.c. */ -double _Py_atanh(double x); -#define m_atanh _Py_atanh -#endif + To save fiddling with configure tests and platform checks, we handle the + special case of zero input directly on all platforms. + */ + if (x == 0.0) { + return x; + } + else { + return log1p(x); + } +} -#ifdef HAVE_EXPM1 -# define m_expm1 expm1 -#else -/* if the system doesn't have expm1, use the substitute - function defined in Modules/_math.c. */ -double _Py_expm1(double x); -#define m_expm1 _Py_expm1 -#endif - -double _Py_log1p(double x); - -/* Use the substitute from _math.c on all platforms: - it includes workarounds for buggy handling of zeros. */ #define m_log1p _Py_log1p diff --git a/Modules/cmathmodule.c b/Modules/cmathmodule.c index 0e0489c5fe2d7..281d3937e2656 100644 --- a/Modules/cmathmodule.c +++ b/Modules/cmathmodule.c @@ -8,11 +8,13 @@ #include "Python.h" #include "pycore_dtoa.h" -#include "_math.h" /* we need DBL_MAX, DBL_MIN, DBL_EPSILON, DBL_MANT_DIG and FLT_RADIX from float.h. We assume that FLT_RADIX is either 2 or 16. */ #include +/* For _Py_log1p with workarounds for buggy handling of zeros. */ +#include "_math.h" + #include "clinic/cmathmodule.c.h" /*[clinic input] module cmath @@ -246,7 +248,7 @@ cmath_acos_impl(PyObject *module, Py_complex z) s2.imag = z.imag; s2 = cmath_sqrt_impl(module, s2); r.real = 2.*atan2(s1.real, s2.real); - r.imag = m_asinh(s2.real*s1.imag - s2.imag*s1.real); + r.imag = asinh(s2.real*s1.imag - s2.imag*s1.real); } errno = 0; return r; @@ -280,7 +282,7 @@ cmath_acosh_impl(PyObject *module, Py_complex z) s2.real = z.real + 1.; s2.imag = z.imag; s2 = cmath_sqrt_impl(module, s2); - r.real = m_asinh(s1.real*s2.real + s1.imag*s2.imag); + r.real = asinh(s1.real*s2.real + s1.imag*s2.imag); r.imag = 2.*atan2(s1.imag, s2.real); } errno = 0; @@ -340,7 +342,7 @@ cmath_asinh_impl(PyObject *module, Py_complex z) s2.real = 1.-z.imag; s2.imag = z.real; s2 = cmath_sqrt_impl(module, s2); - r.real = m_asinh(s1.real*s2.imag-s2.real*s1.imag); + r.real = asinh(s1.real*s2.imag-s2.real*s1.imag); r.imag = atan2(z.imag, s1.real*s2.real-s1.imag*s2.imag); } errno = 0; diff --git a/Modules/mathmodule.c b/Modules/mathmodule.c index 6c12a4e70ddf2..67669f19bc28c 100644 --- a/Modules/mathmodule.c +++ b/Modules/mathmodule.c @@ -61,6 +61,9 @@ raised for division by zero and mod by zero. #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_dtoa.h" // _Py_dg_infinity() #include "pycore_long.h" // _PyLong_GetZero() +/* For DBL_EPSILON in _math.h */ +#include +/* For _Py_log1p with workarounds for buggy handling of zeros. */ #include "_math.h" #include "clinic/mathmodule.c.h" @@ -1166,14 +1169,14 @@ FUNC1(acos, acos, 0, "acos($module, x, /)\n--\n\n" "Return the arc cosine (measured in radians) of x.\n\n" "The result is between 0 and pi.") -FUNC1(acosh, m_acosh, 0, +FUNC1(acosh, acosh, 0, "acosh($module, x, /)\n--\n\n" "Return the inverse hyperbolic cosine of x.") FUNC1(asin, asin, 0, "asin($module, x, /)\n--\n\n" "Return the arc sine (measured in radians) of x.\n\n" "The result is between -pi/2 and pi/2.") -FUNC1(asinh, m_asinh, 0, +FUNC1(asinh, asinh, 0, "asinh($module, x, /)\n--\n\n" "Return the inverse hyperbolic sine of x.") FUNC1(atan, atan, 0, @@ -1184,7 +1187,7 @@ FUNC2(atan2, m_atan2, "atan2($module, y, x, /)\n--\n\n" "Return the arc tangent (measured in radians) of y/x.\n\n" "Unlike atan(y/x), the signs of both x and y are considered.") -FUNC1(atanh, m_atanh, 0, +FUNC1(atanh, atanh, 0, "atanh($module, x, /)\n--\n\n" "Return the inverse hyperbolic tangent of x.") FUNC1(cbrt, cbrt, 0, @@ -1245,7 +1248,7 @@ FUNC1A(erfc, m_erfc, FUNC1(exp, exp, 1, "exp($module, x, /)\n--\n\n" "Return e raised to the power of x.") -FUNC1(expm1, m_expm1, 1, +FUNC1(expm1, expm1, 1, "expm1($module, x, /)\n--\n\n" "Return exp(x)-1.\n\n" "This function avoids the loss of precision involved in the direct " diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index 1b484be5e7a2f..646bc8e4800b1 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -332,7 +332,6 @@ - diff --git a/PCbuild/pythoncore.vcxproj.filters b/PCbuild/pythoncore.vcxproj.filters index c1519760d2207..62aab5bccf9ef 100644 --- a/PCbuild/pythoncore.vcxproj.filters +++ b/PCbuild/pythoncore.vcxproj.filters @@ -689,9 +689,6 @@ Modules - - Modules - Modules diff --git a/configure b/configure index 198b0703fd6af..b72660d24c2a2 100755 --- a/configure +++ b/configure @@ -15092,7 +15092,7 @@ fi LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" -for ac_func in acosh asinh atanh erf erfc expm1 finite gamma +for ac_func in acosh asinh atanh erf erfc expm1 finite gamma lgamma log1p log2 tgamma do : as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" @@ -15101,21 +15101,13 @@ if eval test \"x\$"$as_ac_var"\" = x"yes"; then : #define `$as_echo "HAVE_$ac_func" | $as_tr_cpp` 1 _ACEOF -fi -done - -for ac_func in lgamma log1p log2 tgamma -do : - as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` -ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" -if eval test \"x\$"$as_ac_var"\" = x"yes"; then : - cat >>confdefs.h <<_ACEOF -#define `$as_echo "HAVE_$ac_func" | $as_tr_cpp` 1 -_ACEOF +else + as_fn_error $? "Python requires C99 compatible libm" "$LINENO" 5 fi done +LIBS=$LIBS_SAVE # For multiprocessing module, check that sem_open # actually works. For FreeBSD versions <= 7.2, diff --git a/configure.ac b/configure.ac index edda08daac117..4093938c209a0 100644 --- a/configure.ac +++ b/configure.ac @@ -4692,8 +4692,12 @@ fi LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" -AC_CHECK_FUNCS([acosh asinh atanh erf erfc expm1 finite gamma]) -AC_CHECK_FUNCS([lgamma log1p log2 tgamma]) +AC_CHECK_FUNCS( + [acosh asinh atanh erf erfc expm1 finite gamma lgamma log1p log2 tgamma], + [], + [AC_MSG_ERROR([Python requires C99 compatible libm])] +) +LIBS=$LIBS_SAVE # For multiprocessing module, check that sem_open # actually works. For FreeBSD versions <= 7.2, diff --git a/setup.py b/setup.py index 4bef3cb52f2b2..850cc1ba41eb5 100644 --- a/setup.py +++ b/setup.py @@ -904,18 +904,14 @@ def detect_simple_extensions(self): # Context Variables self.add(Extension('_contextvars', ['_contextvarsmodule.c'])) - shared_math = 'Modules/_math.o' - # math library functions, e.g. sin() self.add(Extension('math', ['mathmodule.c'], - extra_objects=[shared_math], - depends=['_math.h', shared_math], + depends=['_math.h'], libraries=['m'])) # complex math library functions self.add(Extension('cmath', ['cmathmodule.c'], - extra_objects=[shared_math], - depends=['_math.h', shared_math], + depends=['_math.h'], libraries=['m'])) # time libraries: librt may be needed for clock_gettime() From webhook-mailer at python.org Mon Oct 25 12:25:53 2021 From: webhook-mailer at python.org (tiran) Date: Mon, 25 Oct 2021 16:25:53 -0000 Subject: [Python-checkins] bpo-45548: Remove checks for finite and gamma (GH-29206) Message-ID: https://github.com/python/cpython/commit/77e3f224d6ae6d38e5fc899fb5eaadf2b7c255a1 commit: 77e3f224d6ae6d38e5fc899fb5eaadf2b7c255a1 branch: main author: Christian Heimes committer: tiran date: 2021-10-25T18:25:43+02:00 summary: bpo-45548: Remove checks for finite and gamma (GH-29206) files: M configure M configure.ac M pyconfig.h.in diff --git a/configure b/configure index b72660d24c2a2..72463d2271497 100755 --- a/configure +++ b/configure @@ -15092,7 +15092,7 @@ fi LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" -for ac_func in acosh asinh atanh erf erfc expm1 finite gamma lgamma log1p log2 tgamma +for ac_func in acosh asinh atanh erf erfc expm1 log1p log2 do : as_ac_var=`$as_echo "ac_cv_func_$ac_func" | $as_tr_sh` ac_fn_c_check_func "$LINENO" "$ac_func" "$as_ac_var" diff --git a/configure.ac b/configure.ac index 4093938c209a0..5674093fe67a4 100644 --- a/configure.ac +++ b/configure.ac @@ -4693,7 +4693,7 @@ LIBS_SAVE=$LIBS LIBS="$LIBS $LIBM" AC_CHECK_FUNCS( - [acosh asinh atanh erf erfc expm1 finite gamma lgamma log1p log2 tgamma], + [acosh asinh atanh erf erfc expm1 log1p log2], [], [AC_MSG_ERROR([Python requires C99 compatible libm])] ) diff --git a/pyconfig.h.in b/pyconfig.h.in index 081ea61bae834..3e3944a6ec208 100644 --- a/pyconfig.h.in +++ b/pyconfig.h.in @@ -353,9 +353,6 @@ /* Define to 1 if you have the `fexecve' function. */ #undef HAVE_FEXECVE -/* Define to 1 if you have the `finite' function. */ -#undef HAVE_FINITE - /* Define to 1 if you have the `flock' function. */ #undef HAVE_FLOCK @@ -407,9 +404,6 @@ /* Define to 1 if you have the `gai_strerror' function. */ #undef HAVE_GAI_STRERROR -/* Define to 1 if you have the `gamma' function. */ -#undef HAVE_GAMMA - /* Define if we can use gcc inline assembler to get and set mc68881 fpcr */ #undef HAVE_GCC_ASM_FOR_MC68881 @@ -586,9 +580,6 @@ /* Define to 1 if you have the `lchown' function. */ #undef HAVE_LCHOWN -/* Define to 1 if you have the `lgamma' function. */ -#undef HAVE_LGAMMA - /* Define to 1 if you have the `dl' library (-ldl). */ #undef HAVE_LIBDL @@ -1238,9 +1229,6 @@ /* Define to 1 if you have the header file. */ #undef HAVE_TERM_H -/* Define to 1 if you have the `tgamma' function. */ -#undef HAVE_TGAMMA - /* Define to 1 if you have the `timegm' function. */ #undef HAVE_TIMEGM From webhook-mailer at python.org Mon Oct 25 14:01:13 2021 From: webhook-mailer at python.org (tiran) Date: Mon, 25 Oct 2021 18:01:13 -0000 Subject: [Python-checkins] bpo-45595: Make extensions depend on header files (GH-29198) Message-ID: https://github.com/python/cpython/commit/81669d1b7bd668d1bd0fab92836e8b5e7976e362 commit: 81669d1b7bd668d1bd0fab92836e8b5e7976e362 branch: main author: Christian Heimes committer: tiran date: 2021-10-25T20:01:03+02:00 summary: bpo-45595: Make extensions depend on header files (GH-29198) ``setup.py`` and ``makesetup`` now track build dependencies on all Python header files and module specific header files. Signed-off-by: Christian Heimes files: A Misc/NEWS.d/next/Build/2021-10-24-11-02-43.bpo-45595.WI_5YU.rst M Makefile.pre.in M Modules/makesetup M setup.py diff --git a/Makefile.pre.in b/Makefile.pre.in index 4ee8fddbe14ee..7d403663cec0b 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -2117,6 +2117,28 @@ Python/thread.o: @THREADHEADERS@ $(srcdir)/Python/condvar.h .PHONY: build_all_generate_profile build_all_merge_profile .PHONY: gdbhooks +########################################################################## +# Module dependencies + +MODULE_CMATH_DEPS=$(srcdir)/Modules/_math.h $(srcdir)/Modules/_math.c +MODULE_MATH_DEPS=$(srcdir)/Modules/_math.h $(srcdir)/Modules/_math.c +MODULE_PYEXPAT_DEPS=$(srcdir)/Modules/expat/ascii.h $(srcdir)/Modules/expat/asciitab.h $(srcdir)/Modules/expat/expat.h $(srcdir)/Modules/expat/expat_config.h $(srcdir)/Modules/expat/expat_external.h $(srcdir)/Modules/expat/internal.h $(srcdir)/Modules/expat/latin1tab.h $(srcdir)/Modules/expat/utf8tab.h $(srcdir)/Modules/expat/xmlrole.h $(srcdir)/Modules/expat/xmltok.h $(srcdir)/Modules/expat/xmltok_impl.h +MODULE_UNICODEDATA_DEPS=$(srcdir)/Modules/unicodedata_db.h $(srcdir)/Modules/unicodename_db.h +MODULE__BLAKE2_DEPS=$(srcdir)/Modules/_blake2/impl/blake2-config.h $(srcdir)/Modules/_blake2/impl/blake2-dispatch.c $(srcdir)/Modules/_blake2/impl/blake2-impl.h $(srcdir)/Modules/_blake2/impl/blake2-kat.h $(srcdir)/Modules/_blake2/impl/blake2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2b-ref.c $(srcdir)/Modules/_blake2/impl/blake2b-round.h $(srcdir)/Modules/_blake2/impl/blake2b-test.c $(srcdir)/Modules/_blake2/impl/blake2b.c $(srcdir)/Modules/_blake2/impl/blake2bp-test.c $(srcdir)/Modules/_blake2/impl/blake2bp.c $(srcdir)/Modules/_blake2/impl/blake2s-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2s-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2s-load-xop.h $(srcdir)/Modules/_blake2/impl/blake2s-ref.c $(srcdir)/Modules/_blake2/impl/blake2s-round.h $(srcdir)/Modules/_blake2/impl/blake2s-test.c $(srcdir)/Modules/_blake2/impl/blake2s.c $(srcdir)/Modules/_blake2/impl/blake2sp-test.c $(srcdir)/Modules/_blake2/impl/blake2sp.c $(srcdir)/Modules/hashlib.h +MODULE__CTYPES_DEPS=$(srcdir)/Modules/_ctypes/ctypes.h +MODULE__DECIMAL_DEPS=$(srcdir)/Modules/_decimal/docstrings.h $(srcdir)/Modules/_decimal/libmpdec/basearith.h $(srcdir)/Modules/_decimal/libmpdec/bits.h $(srcdir)/Modules/_decimal/libmpdec/constants.h $(srcdir)/Modules/_decimal/libmpdec/convolute.h $(srcdir)/Modules/_decimal/libmpdec/crt.h $(srcdir)/Modules/_decimal/libmpdec/difradix2.h $(srcdir)/Modules/_decimal/libmpdec/fnt.h $(srcdir)/Modules/_decimal/libmpdec/fourstep.h $(srcdir)/Modules/_decimal/libmpdec/io.h $(srcdir)/Modules/_decimal/libmpdec/mpalloc.h $(srcdir)/Modules/_decimal/libmpdec/mpdecimal.h $(srcdir)/Modules/_decimal/libmpdec/numbertheory.h $(srcdir)/Modules/_decimal/libmpdec/sixstep.h $(srcdir)/Modules/_decimal/libmpdec/transpose.h $(srcdir)/Modules/_decimal/libmpdec/typearith.h $(srcdir)/Modules/_decimal/libmpdec/umodarith.h +MODULE__ELEMENTTREE_DEPS=$(srcdir)/Modules/expat/ascii.h $(srcdir)/Modules/expat/asciitab.h $(srcdir)/Modules/expat/expat.h $(srcdir)/Modules/expat/expat_config.h $(srcdir)/Modules/expat/expat_external.h $(srcdir)/Modules/expat/internal.h $(srcdir)/Modules/expat/latin1tab.h $(srcdir)/Modules/expat/utf8tab.h $(srcdir)/Modules/expat/xmlparse.c $(srcdir)/Modules/expat/xmlrole.c $(srcdir)/Modules/expat/xmlrole.h $(srcdir)/Modules/expat/xmltok.c $(srcdir)/Modules/expat/xmltok.h $(srcdir)/Modules/expat/xmltok_impl.h $(srcdir)/Modules/pyexpat.c +MODULE__HASHLIB_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__IO_DEPS=$(srcdir)/Modules/_io/_iomodule.h +MODULE__MD5_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__SHA1_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__SHA256_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__SHA3_DEPS=$(srcdir)/Modules/_sha3/kcp/KeccakHash.c $(srcdir)/Modules/_sha3/kcp/KeccakHash.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-64.macros $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP-opt32.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP-opt64.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-SnP.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-inplace32BI.c $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-opt64-config.h $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-opt64.c $(srcdir)/Modules/_sha3/kcp/KeccakP-1600-unrolling.macros $(srcdir)/Modules/_sha3/kcp/KeccakSponge.c $(srcdir)/Modules/_sha3/kcp/KeccakSponge.h $(srcdir)/Modules/_sha3/kcp/KeccakSponge.inc $(srcdir)/Modules/_sha3/kcp/PlSnP-Fallback.inc $(srcdir)/Modules/_sha3/kcp/SnP-Relaned.h $(srcdir)/Modules/_sha3/kcp/align.h $(srcdir)/Modules/hashlib.h +MODULE__SHA512_DEPS=$(srcdir)/Modules/hashlib.h +MODULE__SOCKET_DEPS=$(srcdir)/Modules/socketmodule.h +MODULE__SSL_DEPS=$(srcdir)/Modules/_ssl.h $(srcdir)/Modules/_ssl/cert.c $(srcdir)/Modules/_ssl/debughelpers.c $(srcdir)/Modules/_ssl/misc.c $(srcdir)/Modules/_ssl_data.h $(srcdir)/Modules/_ssl_data_111.h $(srcdir)/Modules/_ssl_data_300.h $(srcdir)/Modules/socketmodule.h +MODULE__TESTCAPI_DEPS=$(srcdir)/Modules/testcapi_long.h + # IF YOU PUT ANYTHING HERE IT WILL GO AWAY # Local Variables: # mode: makefile diff --git a/Misc/NEWS.d/next/Build/2021-10-24-11-02-43.bpo-45595.WI_5YU.rst b/Misc/NEWS.d/next/Build/2021-10-24-11-02-43.bpo-45595.WI_5YU.rst new file mode 100644 index 0000000000000..bb09d6ead08d4 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-24-11-02-43.bpo-45595.WI_5YU.rst @@ -0,0 +1,2 @@ +``setup.py`` and ``makesetup`` now track build dependencies on all Python +header files and module specific header files. diff --git a/Modules/makesetup b/Modules/makesetup index 1a767838c92be..849f15fddaab0 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -235,7 +235,8 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | *) cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac - rule="$obj: $src; $cc $cpps -c $src -o $obj" + mods_upper=$(echo $mods | tr '[a-z]' '[A-Z]') + rule="$obj: $src \$(MODULE_${mods_upper}_DEPS) \$(PYTHON_HEADERS); $cc $cpps -c \$< -o \$@" echo "$rule" >>$rulesf done case $doconfig in @@ -248,7 +249,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | no) SHAREDMODS="$SHAREDMODS $file";; esac rule="$file: $objs" - rule="$rule; \$(BLDSHARED) $objs $libs $ExtraLibs -o $file" + rule="$rule; \$(BLDSHARED) $objs $libs $ExtraLibs -o \$@" echo "$rule" >>$rulesf done done diff --git a/setup.py b/setup.py index 850cc1ba41eb5..f32dd4c6e5e14 100644 --- a/setup.py +++ b/setup.py @@ -409,8 +409,11 @@ def update_sources_depends(self): for filename in self.distribution.scripts] # Python header files + include_dir = escape(sysconfig.get_path('include')) headers = [sysconfig.get_config_h_filename()] - headers += glob(os.path.join(escape(sysconfig.get_path('include')), "*.h")) + headers.extend(glob(os.path.join(include_dir, "*.h"))) + headers.extend(glob(os.path.join(include_dir, "cpython", "*.h"))) + headers.extend(glob(os.path.join(include_dir, "internal", "*.h"))) for ext in self.extensions: ext.sources = [ find_module_file(filename, moddirlist) @@ -2473,6 +2476,9 @@ def split_var(name, sep): depends=[ 'socketmodule.h', '_ssl.h', + '_ssl_data_111.h', + '_ssl_data_300.h', + '_ssl_data.h', '_ssl/debughelpers.c', '_ssl/misc.c', '_ssl/cert.c', From webhook-mailer at python.org Mon Oct 25 14:02:02 2021 From: webhook-mailer at python.org (tiran) Date: Mon, 25 Oct 2021 18:02:02 -0000 Subject: [Python-checkins] bpo-45548: Add missing extensions to Modules/Setup (GH-29199) Message-ID: https://github.com/python/cpython/commit/ece916e92ce76cfdbfd208605a6fc827db0fbd52 commit: ece916e92ce76cfdbfd208605a6fc827db0fbd52 branch: main author: Christian Heimes committer: tiran date: 2021-10-25T20:01:52+02:00 summary: bpo-45548: Add missing extensions to Modules/Setup (GH-29199) Adds _ctypes, _decimal, _multiprocessing, _posixshmem, _scproxy, _sqlite3, and _uuid. Use Makefile variables to build OpenSSL and TCL/TK modules. Drop ``-lm`` from math lines. Extensions are always linked against libm. Signed-off-by: Christian Heimes files: M Modules/Setup diff --git a/Modules/Setup b/Modules/Setup index 99c7d4e909e18..992ffecb334d5 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -129,6 +129,9 @@ time timemodule.c # Python binary, or need to specify some odd set of compiler switches, # you can uncomment the appropriate lines below. +# To enable all modules for testing, run +# sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.local + # Uncommenting the following line tells makesetup that all following # modules are to be built as shared libraries (see above for more # detail; also note that *static* or *disabled* cancels this effect): @@ -149,12 +152,15 @@ time timemodule.c #_contextvars _contextvarsmodule.c #_csv _csv.c #_datetime _datetimemodule.c +# UNIVERSAL: let mpdecimal.h detect settings +#_decimal -DUNIVERSAL -I$(srcdir)/Modules/_decimal/libmpdec _decimal/_decimal.c _decimal/libmpdec/basearith.c _decimal/libmpdec/constants.c _decimal/libmpdec/context.c _decimal/libmpdec/convolute.c _decimal/libmpdec/crt.c _decimal/libmpdec/difradix2.c _decimal/libmpdec/fnt.c _decimal/libmpdec/fourstep.c _decimal/libmpdec/io.c _decimal/libmpdec/mpalloc.c _decimal/libmpdec/mpdecimal.c _decimal/libmpdec/numbertheory.c _decimal/libmpdec/sixstep.c _decimal/libmpdec/transpose.c #_elementtree -I$(srcdir)/Modules/expat _elementtree.c #_heapq _heapqmodule.c #_json _json.c #_lsprof _lsprof.c rotatingtree.c #_md5 md5module.c #_multibytecodec cjkcodecs/multibytecodec.c +#_multiprocessing -I$(srcdir)/Modules/_multiprocessing _multiprocessing/multiprocessing.c _multiprocessing/semaphore.c #_opcode _opcode.c #_pickle _pickle.c #_posixsubprocess _posixsubprocess.c @@ -171,8 +177,8 @@ time timemodule.c #array arraymodule.c #audioop audioop.c #binascii binascii.c -#cmath cmathmodule.c # -lm -#math mathmodule.c # -lm +#cmath cmathmodule.c +#math mathmodule.c #pyexpat -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c #unicodedata unicodedata.c @@ -180,6 +186,7 @@ time timemodule.c # (If you have a really backward UNIX, select and socket may not be # supported...) +#_posixshmem -I$(srcdir)/Modules/_multiprocessing _multiprocessing/posixshmem.c -lrt #_socket socketmodule.c # socket(2) #fcntl fcntlmodule.c # fcntl(2) and ioctl(2) #grp grpmodule.c # grp(3) @@ -192,18 +199,21 @@ time timemodule.c # Some more UNIX dependent modules -- off by default, since these # are not supported by all UNIX systems: -#_crypt _cryptmodule.c # -lcrypt # crypt(3); breaks many builds. -#nis nismodule.c -lnsl # Sun yellow pages -- not everywhere +#_crypt _cryptmodule.c -lcrypt # crypt(3); breaks many builds. +#nis nismodule.c -I/usr/include/tirpc -lnsl -ltirpc # Sun yellow pages -- not everywhere #termios termios.c # Steen Lumholt's termios module #resource resource.c # Jeremy Hylton's rlimit interface # Modules that require external libraries. #_bz2 _bz2module.c -lbz2 -#_dbm _dbmmodule.c # -lndbm # dbm(3) -#_gdbm -I/usr/local/include -L/usr/local/lib -lgdbm _gdbmmodule.c +#_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -ldl -lffi -DHAVE_FFI_PREP_CIF_VAR -DHAVE_FFI_PREP_CLOSURE_LOC -DHAVE_FFI_CLOSURE_ALLOC +# _dbm _dbmmodule.c # -lndbm # dbm(3) +#_gdbm _gdbmmodule.c -lgdbm #_lzma _lzmamodule.c -llzma -#zlib -I$(prefix)/include -L$(exec_prefix)/lib -lz zlibmodule.c +#_sqlite3 _sqlite/connection.c _sqlite/cursor.c _sqlite/microprotocols.c _sqlite/module.c _sqlite/prepare_protocol.c _sqlite/row.c _sqlite/statement.c _sqlite/util.c -lsqlite3 +#_uuid _uuidmodule.c -luuid +#zlib zlibmodule.c -lz # GNU readline. Unlike previous Python incarnations, GNU readline is # now incorporated in an optional module, configured in the Setup file @@ -215,29 +225,16 @@ time timemodule.c #readline readline.c -lreadline -ltermcap -# Set OpenSSL when not using the system copy found by ./configure. -#OPENSSL=/path/to/openssl/directory - # To dynamically link OpenSSL: -#_ssl _ssl.c \ -# -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -lssl -lcrypto \ -# -DPy_BUILD_CORE_BUILTIN -#_hashlib _hashopenssl.c \ -# -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -lcrypto \ -# -DPy_BUILD_CORE_BUILTIN +#_ssl _ssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) $(OPENSSL_LIBS) +#_hashlib _hashopenssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) -lcrypto # To statically link OpenSSL: -#_ssl _ssl.c \ -# -I$(OPENSSL)/include -L$(OPENSSL)/lib \ +# _ssl _ssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) \ # -l:libssl.a -Wl,--exclude-libs,libssl.a \ -# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ -# -DPy_BUILD_CORE_BUILTIN -#_hashlib _hashopenssl.c \ -# -I$(OPENSSL)/include -L$(OPENSSL)/lib \ -# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a \ -# -DPy_BUILD_CORE_BUILTIN +# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a +# _hashlib _hashopenssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) \ +# -l:libcrypto.a -Wl,--exclude-libs,libcrypto.a # The _tkinter module. # @@ -251,7 +248,7 @@ time timemodule.c # every system. # *** Always uncomment this (leave the leading underscore in!): -# _tkinter _tkinter.c tkappinit.c -DWITH_APPINIT \ +#_tkinter _tkinter.c tkappinit.c -DWITH_APPINIT $(TCLTK_INCLUDES) $(TCLTK_LIBS) \ # *** Uncomment and edit to reflect where your Tcl/Tk libraries are: # -L/usr/local/lib \ # *** Uncomment and edit to reflect where your Tcl/Tk headers are: @@ -269,8 +266,6 @@ time timemodule.c # -DWITH_PIL -I../Extensions/Imaging/libImaging tkImaging.c \ # *** Uncomment and edit for TOGL extension only: # -DWITH_TOGL togl.c \ -# *** Uncomment and edit to reflect your Tcl/Tk versions: -# -ltk8.2 -ltcl8.2 \ # *** Uncomment and edit to reflect where your X11 libraries are: # -L/usr/X11R6/lib \ # *** Or uncomment this for Solaris: @@ -286,11 +281,14 @@ time timemodule.c # provided by the ncurses library. e.g. on Linux, link with -lncurses # instead of -lcurses). -#_curses -lcurses -ltermcap _cursesmodule.c +#_curses -lcurses -lcursesw -ltermcap _cursesmodule.c # Wrapper for the panel library that's part of ncurses and SYSV curses. #_curses_panel -lpanel -lncurses _curses_panel.c +# macOS specific modules +# _scproxy _scproxy.c -framework SystemConfiguration -framework CoreFoundation + # Examples #xx xxmodule.c @@ -302,12 +300,14 @@ xxsubtype xxsubtype.c # Required for the test suite to pass! #_xxsubinterpreters _xxsubinterpretersmodule.c #_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c +#_ctypes_test _ctypes/_ctypes_test.c #_testbuffer _testbuffer.c -#_testcapi _testcapimodule.c # CANNOT be statically compiled! #_testimportmultiple _testimportmultiple.c #_testinternalcapi _testinternalcapi.c #_testmultiphase _testmultiphase.c +#*shared* +#_testcapi _testcapimodule.c # CANNOT be statically compiled! # --- # Uncommenting the following line tells makesetup that all following modules @@ -315,5 +315,5 @@ xxsubtype xxsubtype.c # Required for the test suite to pass! # #*disabled* # -#_sqlite3 _tkinter _curses pyexpat -#_codecs_jp _codecs_kr _codecs_tw unicodedata +# _sqlite3 _tkinter _curses pyexpat +# _codecs_jp _codecs_kr _codecs_tw unicodedata From webhook-mailer at python.org Mon Oct 25 14:53:15 2021 From: webhook-mailer at python.org (tiran) Date: Mon, 25 Oct 2021 18:53:15 -0000 Subject: [Python-checkins] bpo-45548: FreeBSD doesn't like auto vars in makesetup (GH-29216) Message-ID: https://github.com/python/cpython/commit/2b8677a3cd855eb3a579894c64588eab0e006269 commit: 2b8677a3cd855eb3a579894c64588eab0e006269 branch: main author: Christian Heimes committer: tiran date: 2021-10-25T20:52:55+02:00 summary: bpo-45548: FreeBSD doesn't like auto vars in makesetup (GH-29216) files: M Modules/makesetup diff --git a/Modules/makesetup b/Modules/makesetup index 849f15fddaab0..9046657caf4a7 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -236,7 +236,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac mods_upper=$(echo $mods | tr '[a-z]' '[A-Z]') - rule="$obj: $src \$(MODULE_${mods_upper}_DEPS) \$(PYTHON_HEADERS); $cc $cpps -c \$< -o \$@" + rule="$obj: $src \$(MODULE_${mods_upper}_DEPS) \$(PYTHON_HEADERS); $cc $cpps -c $src -o $obj" echo "$rule" >>$rulesf done case $doconfig in @@ -249,7 +249,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | no) SHAREDMODS="$SHAREDMODS $file";; esac rule="$file: $objs" - rule="$rule; \$(BLDSHARED) $objs $libs $ExtraLibs -o \$@" + rule="$rule; \$(BLDSHARED) $objs $libs $ExtraLibs -o $file" echo "$rule" >>$rulesf done done From webhook-mailer at python.org Mon Oct 25 17:26:45 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Mon, 25 Oct 2021 21:26:45 -0000 Subject: [Python-checkins] bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997) Message-ID: https://github.com/python/cpython/commit/6afb285ff0790471a6858e44f85d143f07fda70c commit: 6afb285ff0790471a6858e44f85d143f07fda70c branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-25T15:26:41-06:00 summary: bpo-45020: Add tests for the -X "frozen_modules" option. (gh-28997) We hadn't explicitly added any tests for this, so here they are. https://bugs.python.org/issue45020 files: M Lib/test/test_cmd_line.py M Lib/test/test_embed.py M Programs/_testembed.c diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py index 1dc8c45885cbe..86ee27485c964 100644 --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -123,6 +123,21 @@ def run_python(*args): else: self.assertEqual(err, b'') + def test_xoption_frozen_modules(self): + tests = { + ('=on', 'FrozenImporter'), + ('=off', 'SourceFileLoader'), + ('=', 'FrozenImporter'), + ('', 'FrozenImporter'), + } + for raw, expected in tests: + cmd = ['-X', f'frozen_modules{raw}', + #'-c', 'import os; print(os.__spec__.loader.__name__, end="")'] + '-c', 'import os; print(os.__spec__.loader, end="")'] + with self.subTest(raw): + res = assert_python_ok(*cmd) + self.assertRegex(res.out.decode('utf-8'), expected) + def test_run_module(self): # Test expected operation of the '-m' switch # Switch needs an argument diff --git a/Lib/test/test_embed.py b/Lib/test/test_embed.py index 4186f011e2388..7858f68d971cd 100644 --- a/Lib/test/test_embed.py +++ b/Lib/test/test_embed.py @@ -1466,6 +1466,30 @@ def test_get_argc_argv(self): self.run_embedded_interpreter("test_get_argc_argv") # ignore output + def test_init_use_frozen_modules(self): + tests = { + ('=on', 1), + ('=off', 0), + ('=', 1), + ('', 1), + } + for raw, expected in tests: + optval = f'frozen_modules{raw}' + config = { + 'parse_argv': 2, + 'argv': ['-c'], + 'orig_argv': ['./argv0', '-X', optval, '-c', 'pass'], + 'program_name': './argv0', + 'run_command': 'pass\n', + 'use_environment': 1, + 'xoptions': [optval], + 'use_frozen_modules': expected, + } + env = {'TESTFROZEN': raw[1:]} if raw else None + with self.subTest(repr(raw)): + self.check_all_configs("test_init_use_frozen_modules", config, + api=API_PYTHON, env=env) + class SetConfigTests(unittest.TestCase): def test_set_config(self): diff --git a/Programs/_testembed.c b/Programs/_testembed.c index b61fe3461bdc9..773c6c3e9900a 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -1726,6 +1726,44 @@ static int test_get_argc_argv(void) } +static int check_use_frozen_modules(const char *rawval) +{ + wchar_t optval[100]; + if (rawval == NULL) { + wcscpy(optval, L"frozen_modules"); + } + else if (swprintf(optval, 100, L"frozen_modules=%s", rawval) < 0) { + error("rawval is too long"); + return -1; + } + + PyConfig config; + PyConfig_InitPythonConfig(&config); + + config.parse_argv = 1; + + wchar_t* argv[] = { + L"./argv0", + L"-X", + optval, + L"-c", + L"pass", + }; + config_set_argv(&config, Py_ARRAY_LENGTH(argv), argv); + init_from_config_clear(&config); + + dump_config(); + Py_Finalize(); + return 0; +} + +static int test_init_use_frozen_modules(void) +{ + const char *envvar = getenv("TESTFROZEN"); + return check_use_frozen_modules(envvar); +} + + static int test_unicode_id_init(void) { // bpo-42882: Test that _PyUnicode_FromId() works @@ -1912,6 +1950,7 @@ static struct TestCase TestCases[] = { {"test_run_main", test_run_main}, {"test_run_main_loop", test_run_main_loop}, {"test_get_argc_argv", test_get_argc_argv}, + {"test_init_use_frozen_modules", test_init_use_frozen_modules}, // Audit {"test_open_code_hook", test_open_code_hook}, From webhook-mailer at python.org Tue Oct 26 04:58:52 2021 From: webhook-mailer at python.org (tiran) Date: Tue, 26 Oct 2021 08:58:52 -0000 Subject: [Python-checkins] bpo-45548: makesetup improvements (GH-29225) Message-ID: https://github.com/python/cpython/commit/b5ee79494b2e0d484b7cf59f6746010e22567702 commit: b5ee79494b2e0d484b7cf59f6746010e22567702 branch: main author: Christian Heimes committer: tiran date: 2021-10-26T10:58:43+02:00 summary: bpo-45548: makesetup improvements (GH-29225) * record which modules are build as shared extensions * put object files in same directory as source files * remove dependency on deleted _math.c Signed-off-by: Christian Heimes files: M Makefile.pre.in M Modules/makesetup diff --git a/Makefile.pre.in b/Makefile.pre.in index 7d403663cec0b..60acc16beeee3 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -21,6 +21,7 @@ # === Variables set by makesetup === MODBUILT_NAMES= _MODBUILT_NAMES_ +MODSHARED_NAMES= _MODSHARED_NAMES_ MODDISABLED_NAMES= _MODDISABLED_NAMES_ MODOBJS= _MODOBJS_ MODLIBS= _MODLIBS_ @@ -2120,8 +2121,8 @@ Python/thread.o: @THREADHEADERS@ $(srcdir)/Python/condvar.h ########################################################################## # Module dependencies -MODULE_CMATH_DEPS=$(srcdir)/Modules/_math.h $(srcdir)/Modules/_math.c -MODULE_MATH_DEPS=$(srcdir)/Modules/_math.h $(srcdir)/Modules/_math.c +MODULE_CMATH_DEPS=$(srcdir)/Modules/_math.h +MODULE_MATH_DEPS=$(srcdir)/Modules/_math.h MODULE_PYEXPAT_DEPS=$(srcdir)/Modules/expat/ascii.h $(srcdir)/Modules/expat/asciitab.h $(srcdir)/Modules/expat/expat.h $(srcdir)/Modules/expat/expat_config.h $(srcdir)/Modules/expat/expat_external.h $(srcdir)/Modules/expat/internal.h $(srcdir)/Modules/expat/latin1tab.h $(srcdir)/Modules/expat/utf8tab.h $(srcdir)/Modules/expat/xmlrole.h $(srcdir)/Modules/expat/xmltok.h $(srcdir)/Modules/expat/xmltok_impl.h MODULE_UNICODEDATA_DEPS=$(srcdir)/Modules/unicodedata_db.h $(srcdir)/Modules/unicodename_db.h MODULE__BLAKE2_DEPS=$(srcdir)/Modules/_blake2/impl/blake2-config.h $(srcdir)/Modules/_blake2/impl/blake2-dispatch.c $(srcdir)/Modules/_blake2/impl/blake2-impl.h $(srcdir)/Modules/_blake2/impl/blake2-kat.h $(srcdir)/Modules/_blake2/impl/blake2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2b-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2b-ref.c $(srcdir)/Modules/_blake2/impl/blake2b-round.h $(srcdir)/Modules/_blake2/impl/blake2b-test.c $(srcdir)/Modules/_blake2/impl/blake2b.c $(srcdir)/Modules/_blake2/impl/blake2bp-test.c $(srcdir)/Modules/_blake2/impl/blake2bp.c $(srcdir)/Modules/_blake2/impl/blake2s-load-sse2.h $(srcdir)/Modules/_blake2/impl/blake2s-load-sse41.h $(srcdir)/Modules/_blake2/impl/blake2s-load-xop.h $(srcdir)/Modules/_blake2/impl/blake2s-ref.c $(srcdir)/Modules/_blake2/impl/blake2s-round.h $(srcdir)/Modules/_blake2/impl/blake2s-test.c $(srcdir)/Modules/_blake2/impl/blake2s.c $(srcdir)/Modules/_blake2/impl/blake2sp-test.c $(srcdir)/Modules/_blake2/impl/blake2sp.c $(srcdir)/Modules/hashlib.h diff --git a/Modules/makesetup b/Modules/makesetup index 9046657caf4a7..7547315281643 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -31,6 +31,7 @@ # - insert an identifying comment at the start # - replace _MODBUILT_NAMES_ by the list of *static* and *shared* modules # from Setup +# - replace _MODBSHARED_NAMES_ by the list of *shared* modules from Setup # - replace _MODDISABLED_NAMES_ by the list of *disabled* modules from Setup # - replace _MODOBJS_ by the list of objects from Setup (except for # Setup files after a -n option) @@ -114,6 +115,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | " >$rulesf DEFS= BUILT= + BUILT_SHARED= DISABLED= MODS= SHAREDMODS= @@ -222,7 +224,10 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | *.m) obj=`basename $src .m`.o; cc='$(CC)';; # Obj-C *) continue;; esac - obj="$srcdir/$obj" + case $src in + */*) obj="$srcdir/`dirname $src`/$obj";; + *) obj="$srcdir/$obj";; + esac objs="$objs $obj" case $src in glmodule.c) ;; @@ -246,7 +251,10 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | do file="$srcdir/$mod\$(EXT_SUFFIX)" case $doconfig in - no) SHAREDMODS="$SHAREDMODS $file";; + no) + SHAREDMODS="$SHAREDMODS $file" + BUILT_SHARED="$BUILT_SHARED $mod" + ;; esac rule="$file: $objs" rule="$rule; \$(BLDSHARED) $objs $libs $ExtraLibs -o $file" @@ -296,6 +304,7 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | str="# Generated automatically from $makepre by makesetup." echo "$str" >>$sedf echo "s%_MODBUILT_NAMES_%$BUILT%" >>$sedf + echo "s%_MODSHARED_NAMES_%$BUILT_SHARED%" >>$sedf echo "s%_MODDISABLED_NAMES_%$DISABLED%" >>$sedf echo "s%_MODOBJS_%$OBJS%" >>$sedf echo "s%_MODLIBS_%$LIBS%" >>$sedf From webhook-mailer at python.org Tue Oct 26 17:56:52 2021 From: webhook-mailer at python.org (tjguk) Date: Tue, 26 Oct 2021 21:56:52 -0000 Subject: [Python-checkins] bpo-40915: Fix mmap resize bugs on Windows (GH-29213) Message-ID: https://github.com/python/cpython/commit/aea5ecc458084e01534ea6a11f4181f369869082 commit: aea5ecc458084e01534ea6a11f4181f369869082 branch: main author: Tim Golden committer: tjguk date: 2021-10-26T22:56:43+01:00 summary: bpo-40915: Fix mmap resize bugs on Windows (GH-29213) (original patch by eryksun) Correctly hand various failure modes when resizing an mmap on Windows: * Resizing a pagefile-backed mmap now creates a new mmap and copies data * Attempting to resize when another mapping is held on the same file raises an OSError * Attempting to resize a nametagged mmap raises an OSError if another mapping is held with the same nametag files: M Doc/library/mmap.rst M Lib/test/test_mmap.py M Modules/mmapmodule.c diff --git a/Doc/library/mmap.rst b/Doc/library/mmap.rst index da174753361b2..c1ebd80abd977 100644 --- a/Doc/library/mmap.rst +++ b/Doc/library/mmap.rst @@ -256,6 +256,14 @@ To map anonymous memory, -1 should be passed as the fileno along with the length with :const:`ACCESS_READ` or :const:`ACCESS_COPY`, resizing the map will raise a :exc:`TypeError` exception. + **On Windows**: Resizing the map will raise an :exc:`OSError` if there are other + maps against the same named file. Resizing an anonymous map (ie against the + pagefile) will silently create a new map with the original data copied over + up to the length of the new size. + + .. versionchanged:: 3.11 + Correctly fails if attempting to resize when another map is held + Allows resize against an anonymous map on Windows .. method:: rfind(sub[, start[, end]]) diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 8f34c182f82ea..82e2d2adb7dcf 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -5,6 +5,7 @@ import os import re import itertools +import random import socket import sys import weakref @@ -707,7 +708,6 @@ def test_write_returning_the_number_of_bytes_written(self): self.assertEqual(mm.write(b"yz"), 2) self.assertEqual(mm.write(b"python"), 6) - @unittest.skipIf(os.name == 'nt', 'cannot resize anonymous mmaps on Windows') def test_resize_past_pos(self): m = mmap.mmap(-1, 8192) self.addCleanup(m.close) @@ -796,6 +796,80 @@ def test_madvise(self): self.assertEqual(m.madvise(mmap.MADV_NORMAL, 0, 2), None) self.assertEqual(m.madvise(mmap.MADV_NORMAL, 0, size), None) + @unittest.skipUnless(os.name == 'nt', 'requires Windows') + def test_resize_up_when_mapped_to_pagefile(self): + """If the mmap is backed by the pagefile ensure a resize up can happen + and that the original data is still in place + """ + start_size = PAGESIZE + new_size = 2 * start_size + data = bytes(random.getrandbits(8) for _ in range(start_size)) + + m = mmap.mmap(-1, start_size) + m[:] = data + m.resize(new_size) + self.assertEqual(len(m), new_size) + self.assertEqual(m[:start_size], data[:start_size]) + + @unittest.skipUnless(os.name == 'nt', 'requires Windows') + def test_resize_down_when_mapped_to_pagefile(self): + """If the mmap is backed by the pagefile ensure a resize down up can happen + and that a truncated form of the original data is still in place + """ + start_size = PAGESIZE + new_size = start_size // 2 + data = bytes(random.getrandbits(8) for _ in range(start_size)) + + m = mmap.mmap(-1, start_size) + m[:] = data + m.resize(new_size) + self.assertEqual(len(m), new_size) + self.assertEqual(m[:new_size], data[:new_size]) + + @unittest.skipUnless(os.name == 'nt', 'requires Windows') + def test_resize_fails_if_mapping_held_elsewhere(self): + """If more than one mapping is held against a named file on Windows, neither + mapping can be resized + """ + start_size = 2 * PAGESIZE + reduced_size = PAGESIZE + + f = open(TESTFN, 'wb+') + f.truncate(start_size) + try: + m1 = mmap.mmap(f.fileno(), start_size) + m2 = mmap.mmap(f.fileno(), start_size) + with self.assertRaises(OSError): + m1.resize(reduced_size) + with self.assertRaises(OSError): + m2.resize(reduced_size) + m2.close() + m1.resize(reduced_size) + self.assertEqual(m1.size(), reduced_size) + self.assertEqual(os.stat(f.fileno()).st_size, reduced_size) + finally: + f.close() + + @unittest.skipUnless(os.name == 'nt', 'requires Windows') + def test_resize_succeeds_with_error_for_second_named_mapping(self): + """If a more than one mapping exists of the same name, none of them can + be resized: they'll raise an Exception and leave the original mapping intact + """ + start_size = 2 * PAGESIZE + reduced_size = PAGESIZE + tagname = "TEST" + data_length = 8 + data = bytes(random.getrandbits(8) for _ in range(data_length)) + + m1 = mmap.mmap(-1, start_size, tagname=tagname) + m2 = mmap.mmap(-1, start_size, tagname=tagname) + m1[:data_length] = data + self.assertEqual(m2[:data_length], data) + with self.assertRaises(OSError): + m1.resize(reduced_size) + self.assertEqual(m1.size(), start_size) + self.assertEqual(m1[:data_length], data) + self.assertEqual(m2[:data_length], data) class LargeMmapTests(unittest.TestCase): diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index 6397b0d4b8109..dfa10f633bbd5 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -32,6 +32,7 @@ #ifdef MS_WINDOWS #include +#include static int my_getpagesize(void) { @@ -376,14 +377,15 @@ is_resizeable(mmap_object *self) { if (self->exports > 0) { PyErr_SetString(PyExc_BufferError, - "mmap can't resize with extant buffers exported."); + "mmap can't resize with extant buffers exported."); return 0; } if ((self->access == ACCESS_WRITE) || (self->access == ACCESS_DEFAULT)) return 1; PyErr_Format(PyExc_TypeError, - "mmap can't resize a readonly or copy-on-write memory map."); + "mmap can't resize a readonly or copy-on-write memory map."); return 0; + } @@ -503,51 +505,110 @@ mmap_resize_method(mmap_object *self, } { + /* + To resize an mmap on Windows: + + - Close the existing mapping + - If the mapping is backed to a named file: + unmap the view, clear the data, and resize the file + If the file can't be resized (eg because it has other mapped references + to it) then let the mapping be recreated at the original size and set + an error code so an exception will be raised. + - Create a new mapping of the relevant size to the same file + - Map a new view of the resized file + - If the mapping is backed by the pagefile: + copy any previous data into the new mapped area + unmap the original view which will release the memory + */ #ifdef MS_WINDOWS - DWORD dwErrCode = 0; - DWORD off_hi, off_lo, newSizeLow, newSizeHigh; - /* First, unmap the file view */ - UnmapViewOfFile(self->data); - self->data = NULL; - /* Close the mapping object */ + DWORD error = 0, file_resize_error = 0; + char* old_data = self->data; + LARGE_INTEGER offset, max_size; + offset.QuadPart = self->offset; + max_size.QuadPart = self->offset + new_size; + /* close the file mapping */ CloseHandle(self->map_handle); - self->map_handle = NULL; - /* Move to the desired EOF position */ - newSizeHigh = (DWORD)((self->offset + new_size) >> 32); - newSizeLow = (DWORD)((self->offset + new_size) & 0xFFFFFFFF); - off_hi = (DWORD)(self->offset >> 32); - off_lo = (DWORD)(self->offset & 0xFFFFFFFF); - SetFilePointer(self->file_handle, - newSizeLow, &newSizeHigh, FILE_BEGIN); - /* Change the size of the file */ - SetEndOfFile(self->file_handle); - /* Create another mapping object and remap the file view */ + /* if the file mapping still exists, it cannot be resized. */ + if (self->tagname) { + self->map_handle = OpenFileMapping(FILE_MAP_WRITE, FALSE, + self->tagname); + if (self->map_handle) { + PyErr_SetFromWindowsErr(ERROR_USER_MAPPED_FILE); + return NULL; + } + } else { + self->map_handle = NULL; + } + + /* if it's not the paging file, unmap the view and resize the file */ + if (self->file_handle != INVALID_HANDLE_VALUE) { + if (!UnmapViewOfFile(self->data)) { + return PyErr_SetFromWindowsErr(GetLastError()); + }; + self->data = NULL; + /* resize the file */ + if (!SetFilePointerEx(self->file_handle, max_size, NULL, + FILE_BEGIN) || + !SetEndOfFile(self->file_handle)) { + /* resizing failed. try to remap the file */ + file_resize_error = GetLastError(); + new_size = max_size.QuadPart = self->size; + } + } + + /* create a new file mapping and map a new view */ + /* FIXME: call CreateFileMappingW with wchar_t tagname */ self->map_handle = CreateFileMapping( self->file_handle, NULL, PAGE_READWRITE, - 0, - 0, + max_size.HighPart, + max_size.LowPart, self->tagname); - if (self->map_handle != NULL) { - self->data = (char *) MapViewOfFile(self->map_handle, - FILE_MAP_WRITE, - off_hi, - off_lo, - new_size); + + error = GetLastError(); + if (error == ERROR_ALREADY_EXISTS) { + CloseHandle(self->map_handle); + self->map_handle = NULL; + } + else if (self->map_handle != NULL) { + self->data = MapViewOfFile(self->map_handle, + FILE_MAP_WRITE, + offset.HighPart, + offset.LowPart, + new_size); if (self->data != NULL) { + /* copy the old view if using the paging file */ + if (self->file_handle == INVALID_HANDLE_VALUE) { + memcpy(self->data, old_data, + self->size < new_size ? self->size : new_size); + if (!UnmapViewOfFile(old_data)) { + error = GetLastError(); + } + } self->size = new_size; - Py_RETURN_NONE; - } else { - dwErrCode = GetLastError(); + } + else { + error = GetLastError(); CloseHandle(self->map_handle); self->map_handle = NULL; } - } else { - dwErrCode = GetLastError(); } - PyErr_SetFromWindowsErr(dwErrCode); - return NULL; + + if (error) { + return PyErr_SetFromWindowsErr(error); + return NULL; + } + /* It's possible for a resize to fail, typically because another mapping + is still held against the same underlying file. Even if nothing has + failed -- ie we're still returning a valid file mapping -- raise the + error as an exception as the resize won't have happened + */ + if (file_resize_error) { + PyErr_SetFromWindowsErr(file_resize_error); + return NULL; + } + Py_RETURN_NONE; #endif /* MS_WINDOWS */ #ifdef UNIX From webhook-mailer at python.org Tue Oct 26 19:22:43 2021 From: webhook-mailer at python.org (benjaminp) Date: Tue, 26 Oct 2021 23:22:43 -0000 Subject: [Python-checkins] Remove unused variables. (GH-29231) Message-ID: https://github.com/python/cpython/commit/19a6c41e56f129a67e2a3c96464ba893a97236f7 commit: 19a6c41e56f129a67e2a3c96464ba893a97236f7 branch: main author: Benjamin Peterson committer: benjaminp date: 2021-10-26T16:22:34-07:00 summary: Remove unused variables. (GH-29231) files: M Modules/_io/bufferedio.c diff --git a/Modules/_io/bufferedio.c b/Modules/_io/bufferedio.c index dc6371a572b9f..28995051abd4b 100644 --- a/Modules/_io/bufferedio.c +++ b/Modules/_io/bufferedio.c @@ -1073,7 +1073,7 @@ _buffered_readline(buffered *self, Py_ssize_t limit) { PyObject *res = NULL; PyObject *chunks = NULL; - Py_ssize_t n, written = 0; + Py_ssize_t n; const char *start, *s, *end; CHECK_CLOSED(self, "readline of closed file") @@ -1115,7 +1115,6 @@ _buffered_readline(buffered *self, Py_ssize_t limit) goto end; } Py_CLEAR(res); - written += n; self->pos += n; if (limit >= 0) limit -= n; @@ -1160,7 +1159,6 @@ _buffered_readline(buffered *self, Py_ssize_t limit) goto end; } Py_CLEAR(res); - written += n; if (limit >= 0) limit -= n; } @@ -1861,7 +1859,6 @@ _bufferedwriter_raw_write(buffered *self, char *start, Py_ssize_t len) static PyObject * _bufferedwriter_flush_unlocked(buffered *self) { - Py_ssize_t written = 0; Py_off_t n, rewind; if (!VALID_WRITE_BUFFER(self) || self->write_pos == self->write_end) @@ -1890,7 +1887,6 @@ _bufferedwriter_flush_unlocked(buffered *self) } self->write_pos += n; self->raw_pos = self->write_pos; - written += Py_SAFE_DOWNCAST(n, Py_off_t, Py_ssize_t); /* Partial writes can return successfully when interrupted by a signal (see write(2)). We must run signal handlers before blocking another time, possibly indefinitely. */ From webhook-mailer at python.org Wed Oct 27 05:45:53 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 27 Oct 2021 09:45:53 -0000 Subject: [Python-checkins] bpo-44511: Improve the bytecode for class and mapping patterns (GH-26922) Message-ID: https://github.com/python/cpython/commit/82a662e5216a9b3969054c540a759a9493468510 commit: 82a662e5216a9b3969054c540a759a9493468510 branch: main author: Brandt Bucher committer: markshannon date: 2021-10-27T10:45:35+01:00 summary: bpo-44511: Improve the bytecode for class and mapping patterns (GH-26922) * Refactor mapping patterns and speed up class patterns. * Simplify MATCH_KEYS and MATCH_CLASS. * Add COPY opcode. files: A Misc/NEWS.d/next/Core and Builtins/2021-06-26-16-55-08.bpo-44511.k8sMvV.rst A Programs/_freeze_importlib M Doc/library/dis.rst M Doc/whatsnew/3.11.rst M Include/opcode.h M Lib/importlib/_bootstrap_external.py M Lib/opcode.py M Python/ceval.c M Python/compile.c M Python/opcode_targets.h diff --git a/Doc/library/dis.rst b/Doc/library/dis.rst index 9958dea5587f1..85cc4afb7e48c 100644 --- a/Doc/library/dis.rst +++ b/Doc/library/dis.rst @@ -755,15 +755,6 @@ iterations of the loop. .. versionadded:: 3.11 -.. opcode:: COPY_DICT_WITHOUT_KEYS - - TOS is a tuple of mapping keys, and TOS1 is the match subject. Replace TOS - with a :class:`dict` formed from the items of TOS1, but without any of the - keys in TOS. - - .. versionadded:: 3.10 - - .. opcode:: GET_LEN Push ``len(TOS)`` onto the stack. @@ -795,11 +786,14 @@ iterations of the loop. TOS is a tuple of mapping keys, and TOS1 is the match subject. If TOS1 contains all of the keys in TOS, push a :class:`tuple` containing the - corresponding values, followed by ``True``. Otherwise, push ``None``, - followed by ``False``. + corresponding values. Otherwise, push ``None``. .. versionadded:: 3.10 + .. versionchanged:: 3.11 + Previously, this instruction also pushed a boolean value indicating + success (``True``) or failure (``False``). + All of the following opcodes use their arguments. @@ -1277,12 +1271,16 @@ All of the following opcodes use their arguments. against, and TOS2 is the match subject. *count* is the number of positional sub-patterns. - Pop TOS. If TOS2 is an instance of TOS1 and has the positional and keyword - attributes required by *count* and TOS, set TOS to ``True`` and TOS1 to a - tuple of extracted attributes. Otherwise, set TOS to ``False``. + Pop TOS, TOS1, and TOS2. If TOS2 is an instance of TOS1 and has the + positional and keyword attributes required by *count* and TOS, push a tuple + of extracted attributes. Otherwise, push ``None``. .. versionadded:: 3.10 + .. versionchanged:: 3.11 + Previously, this instruction also pushed a boolean value indicating + success (``True``) or failure (``False``). + .. opcode:: GEN_START (kind) Pops TOS. If TOS was not ``None``, raises an exception. The ``kind`` @@ -1301,6 +1299,14 @@ All of the following opcodes use their arguments. .. versionadded:: 3.10 +.. opcode:: COPY (i) + + Push the *i*-th item to the top of the stack. The item is not removed from its + original location. + + .. versionadded:: 3.11 + + .. opcode:: HAVE_ARGUMENT This is not really an opcode. It identifies the dividing line between diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 7b3ce9bc7feaf..21ad4669476ef 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -308,6 +308,16 @@ CPython bytecode changes fashion as :opcode:`CALL_METHOD`, but also supports keyword arguments. Works in tandem with :opcode:`LOAD_METHOD`. +* Removed ``COPY_DICT_WITHOUT_KEYS``. + +* :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` no longer push an additional + boolean value indicating whether the match succeeded or failed. Instead, they + indicate failure with :const:`None` (where a tuple of extracted values would + otherwise be). + +* Added :opcode:`COPY`, which pushes the *i*-th item to the top of the stack. + The item is not removed from its original location. + Deprecated ========== diff --git a/Include/opcode.h b/Include/opcode.h index f8c02b840e052..87ed32c019909 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -34,7 +34,6 @@ extern "C" { #define MATCH_MAPPING 31 #define MATCH_SEQUENCE 32 #define MATCH_KEYS 33 -#define COPY_DICT_WITHOUT_KEYS 34 #define PUSH_EXC_INFO 35 #define POP_EXCEPT_AND_RERAISE 37 #define WITH_EXCEPT_START 49 @@ -104,6 +103,7 @@ extern "C" { #define IS_OP 117 #define CONTAINS_OP 118 #define RERAISE 119 +#define COPY 120 #define JUMP_IF_NOT_EXC_MATCH 121 #define LOAD_FAST 124 #define STORE_FAST 125 @@ -142,24 +142,24 @@ extern "C" { #define BINARY_ADD_UNICODE 14 #define BINARY_ADD_UNICODE_INPLACE_FAST 18 #define BINARY_MULTIPLY_ADAPTIVE 21 -#define BINARY_MULTIPLY_INT 36 -#define BINARY_MULTIPLY_FLOAT 38 -#define BINARY_SUBSCR_ADAPTIVE 39 -#define BINARY_SUBSCR_LIST_INT 40 -#define BINARY_SUBSCR_TUPLE_INT 41 -#define BINARY_SUBSCR_DICT 42 -#define CALL_FUNCTION_ADAPTIVE 43 -#define CALL_FUNCTION_BUILTIN_O 44 -#define CALL_FUNCTION_BUILTIN_FAST 45 -#define CALL_FUNCTION_LEN 46 -#define CALL_FUNCTION_ISINSTANCE 47 -#define CALL_FUNCTION_PY_SIMPLE 48 -#define JUMP_ABSOLUTE_QUICK 58 -#define LOAD_ATTR_ADAPTIVE 80 -#define LOAD_ATTR_INSTANCE_VALUE 81 -#define LOAD_ATTR_WITH_HINT 87 -#define LOAD_ATTR_SLOT 88 -#define LOAD_ATTR_MODULE 120 +#define BINARY_MULTIPLY_INT 34 +#define BINARY_MULTIPLY_FLOAT 36 +#define BINARY_SUBSCR_ADAPTIVE 38 +#define BINARY_SUBSCR_LIST_INT 39 +#define BINARY_SUBSCR_TUPLE_INT 40 +#define BINARY_SUBSCR_DICT 41 +#define CALL_FUNCTION_ADAPTIVE 42 +#define CALL_FUNCTION_BUILTIN_O 43 +#define CALL_FUNCTION_BUILTIN_FAST 44 +#define CALL_FUNCTION_LEN 45 +#define CALL_FUNCTION_ISINSTANCE 46 +#define CALL_FUNCTION_PY_SIMPLE 47 +#define JUMP_ABSOLUTE_QUICK 48 +#define LOAD_ATTR_ADAPTIVE 58 +#define LOAD_ATTR_INSTANCE_VALUE 80 +#define LOAD_ATTR_WITH_HINT 81 +#define LOAD_ATTR_SLOT 87 +#define LOAD_ATTR_MODULE 88 #define LOAD_GLOBAL_ADAPTIVE 122 #define LOAD_GLOBAL_MODULE 123 #define LOAD_GLOBAL_BUILTIN 127 diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index ef4f23a4b499f..4edcf9a3ce0cb 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -364,6 +364,8 @@ def _write_atomic(path, data, mode=0o666): # Python 3.11a1 3459 (PEP 657: add end line numbers and column offsets for instructions) # Python 3.11a1 3460 (Add co_qualname field to PyCodeObject bpo-44530) # Python 3.11a1 3461 (JUMP_ABSOLUTE must jump backwards) +# Python 3.11a2 3462 (bpo-44511: remove COPY_DICT_WITHOUT_KEYS, change +# MATCH_CLASS and MATCH_KEYS, and add COPY) # # MAGIC must change whenever the bytecode emitted by the compiler may no @@ -373,7 +375,7 @@ def _write_atomic(path, data, mode=0o666): # Whenever MAGIC_NUMBER is changed, the ranges in the magic_values array # in PC/launcher.c must also be updated. -MAGIC_NUMBER = (3461).to_bytes(2, 'little') + b'\r\n' +MAGIC_NUMBER = (3462).to_bytes(2, 'little') + b'\r\n' _RAW_MAGIC_NUMBER = int.from_bytes(MAGIC_NUMBER, 'little') # For import.c _PYCACHE = '__pycache__' diff --git a/Lib/opcode.py b/Lib/opcode.py index 5377ec32bf153..66d5ca766bec6 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -85,7 +85,7 @@ def jabs_op(name, op): def_op('MATCH_MAPPING', 31) def_op('MATCH_SEQUENCE', 32) def_op('MATCH_KEYS', 33) -def_op('COPY_DICT_WITHOUT_KEYS', 34) + def_op('PUSH_EXC_INFO', 35) def_op('POP_EXCEPT_AND_RERAISE', 37) @@ -165,7 +165,7 @@ def jabs_op(name, op): def_op('IS_OP', 117) def_op('CONTAINS_OP', 118) def_op('RERAISE', 119) - +def_op('COPY', 120) jabs_op('JUMP_IF_NOT_EXC_MATCH', 121) def_op('LOAD_FAST', 124) # Local variable number diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-26-16-55-08.bpo-44511.k8sMvV.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-26-16-55-08.bpo-44511.k8sMvV.rst new file mode 100644 index 0000000000000..352e046126c99 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-06-26-16-55-08.bpo-44511.k8sMvV.rst @@ -0,0 +1 @@ +Improve the generated bytecode for class and mapping patterns. diff --git a/Programs/_freeze_importlib b/Programs/_freeze_importlib new file mode 100755 index 0000000000000..817856481b449 Binary files /dev/null and b/Programs/_freeze_importlib differ diff --git a/Python/ceval.c b/Python/ceval.c index adc7b536247b2..a0f4c801e8f8c 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4143,25 +4143,30 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr } TARGET(MATCH_CLASS) { - // Pop TOS. On success, set TOS to True and TOS1 to a tuple of - // attributes. On failure, set TOS to False. + // Pop TOS and TOS1. Set TOS to a tuple of attributes on success, or + // None on failure. PyObject *names = POP(); - PyObject *type = TOP(); - PyObject *subject = SECOND(); + PyObject *type = POP(); + PyObject *subject = TOP(); assert(PyTuple_CheckExact(names)); PyObject *attrs = match_class(tstate, subject, type, oparg, names); Py_DECREF(names); + Py_DECREF(type); if (attrs) { // Success! assert(PyTuple_CheckExact(attrs)); - Py_DECREF(subject); - SET_SECOND(attrs); + SET_TOP(attrs); } else if (_PyErr_Occurred(tstate)) { + // Error! goto error; } - Py_DECREF(type); - SET_TOP(PyBool_FromLong(!!attrs)); + else { + // Failure! + Py_INCREF(Py_None); + SET_TOP(Py_None); + } + Py_DECREF(subject); DISPATCH(); } @@ -4171,6 +4176,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *res = match ? Py_True : Py_False; Py_INCREF(res); PUSH(res); + PREDICT(POP_JUMP_IF_FALSE); DISPATCH(); } @@ -4180,12 +4186,12 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr PyObject *res = match ? Py_True : Py_False; Py_INCREF(res); PUSH(res); + PREDICT(POP_JUMP_IF_FALSE); DISPATCH(); } TARGET(MATCH_KEYS) { - // On successful match for all keys, PUSH(values) and PUSH(True). - // Otherwise, PUSH(None) and PUSH(False). + // On successful match, PUSH(values). Otherwise, PUSH(None). PyObject *keys = TOP(); PyObject *subject = SECOND(); PyObject *values_or_none = match_keys(tstate, subject, keys); @@ -4193,40 +4199,6 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } PUSH(values_or_none); - if (Py_IsNone(values_or_none)) { - Py_INCREF(Py_False); - PUSH(Py_False); - DISPATCH(); - } - assert(PyTuple_CheckExact(values_or_none)); - Py_INCREF(Py_True); - PUSH(Py_True); - DISPATCH(); - } - - TARGET(COPY_DICT_WITHOUT_KEYS) { - // rest = dict(TOS1) - // for key in TOS: - // del rest[key] - // SET_TOP(rest) - PyObject *keys = TOP(); - PyObject *subject = SECOND(); - PyObject *rest = PyDict_New(); - if (rest == NULL || PyDict_Update(rest, subject)) { - Py_XDECREF(rest); - goto error; - } - // This may seem a bit inefficient, but keys is rarely big enough to - // actually impact runtime. - assert(PyTuple_CheckExact(keys)); - for (Py_ssize_t i = 0; i < PyTuple_GET_SIZE(keys); i++) { - if (PyDict_DelItem(rest, PyTuple_GET_ITEM(keys, i))) { - Py_DECREF(rest); - goto error; - } - } - Py_DECREF(keys); - SET_TOP(rest); DISPATCH(); } @@ -5027,6 +4999,14 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr DISPATCH(); } + TARGET(COPY) { + assert(oparg != 0); + PyObject *peek = PEEK(oparg); + Py_INCREF(peek); + PUSH(peek); + DISPATCH(); + } + TARGET(EXTENDED_ARG) { int oldoparg = oparg; NEXTOPARG(); diff --git a/Python/compile.c b/Python/compile.c index 0c025acec1491..28b5c076d50aa 100644 --- a/Python/compile.c +++ b/Python/compile.c @@ -1248,18 +1248,17 @@ stack_effect(int opcode, int oparg, int jump) case DICT_MERGE: case DICT_UPDATE: return -1; - case COPY_DICT_WITHOUT_KEYS: - return 0; case MATCH_CLASS: - return -1; + return -2; case GET_LEN: case MATCH_MAPPING: case MATCH_SEQUENCE: - return 1; case MATCH_KEYS: - return 2; + return 1; case ROT_N: return 0; + case COPY: + return 1; default: return PY_INVALID_STACK_EFFECT; } @@ -6118,10 +6117,16 @@ compiler_pattern_class(struct compiler *c, pattern_ty p, pattern_context *pc) } ADDOP_LOAD_CONST_NEW(c, attr_names); ADDOP_I(c, MATCH_CLASS, nargs); - // TOS is now a tuple of (nargs + nattrs) attributes. Preserve it: + ADDOP(c, DUP_TOP); + ADDOP_LOAD_CONST(c, Py_None); + ADDOP_I(c, IS_OP, 1); + // TOS is now a tuple of (nargs + nattrs) attributes (or None): pc->on_top++; RETURN_IF_FALSE(jump_to_fail_pop(c, pc, POP_JUMP_IF_FALSE)); + ADDOP_I(c, UNPACK_SEQUENCE, nargs + nattrs); + pc->on_top += nargs + nattrs - 1; for (i = 0; i < nargs + nattrs; i++) { + pc->on_top--; pattern_ty pattern; if (i < nargs) { // Positional: @@ -6132,17 +6137,12 @@ compiler_pattern_class(struct compiler *c, pattern_ty p, pattern_context *pc) pattern = asdl_seq_GET(kwd_patterns, i - nargs); } if (WILDCARD_CHECK(pattern)) { + ADDOP(c, POP_TOP); continue; } - // Get the i-th attribute, and match it against the i-th pattern: - ADDOP(c, DUP_TOP); - ADDOP_LOAD_CONST_NEW(c, PyLong_FromSsize_t(i)); - ADDOP(c, BINARY_SUBSCR); RETURN_IF_FALSE(compiler_pattern_subpattern(c, pattern, pc)); } // Success! Pop the tuple of attributes: - pc->on_top--; - ADDOP(c, POP_TOP); return 1; } @@ -6183,7 +6183,7 @@ compiler_pattern_mapping(struct compiler *c, pattern_ty p, pattern_context *pc) return compiler_error(c, "too many sub-patterns in mapping pattern"); } // Collect all of the keys into a tuple for MATCH_KEYS and - // COPY_DICT_WITHOUT_KEYS. They can either be dotted names or literals: + // **rest. They can either be dotted names or literals: // Maintaining a set of Constant_kind kind keys allows us to raise a // SyntaxError in the case of duplicates. @@ -6235,35 +6235,45 @@ compiler_pattern_mapping(struct compiler *c, pattern_ty p, pattern_context *pc) ADDOP(c, MATCH_KEYS); // There's now a tuple of keys and a tuple of values on top of the subject: pc->on_top += 2; + ADDOP(c, DUP_TOP); + ADDOP_LOAD_CONST(c, Py_None); + ADDOP_I(c, IS_OP, 1); RETURN_IF_FALSE(jump_to_fail_pop(c, pc, POP_JUMP_IF_FALSE)); // So far so good. Use that tuple of values on the stack to match // sub-patterns against: + ADDOP_I(c, UNPACK_SEQUENCE, size); + pc->on_top += size - 1; for (Py_ssize_t i = 0; i < size; i++) { + pc->on_top--; pattern_ty pattern = asdl_seq_GET(patterns, i); - if (WILDCARD_CHECK(pattern)) { - continue; - } - ADDOP(c, DUP_TOP); - ADDOP_LOAD_CONST_NEW(c, PyLong_FromSsize_t(i)); - ADDOP(c, BINARY_SUBSCR); RETURN_IF_FALSE(compiler_pattern_subpattern(c, pattern, pc)); } - // If we get this far, it's a match! We're done with the tuple of values, - // and whatever happens next should consume the tuple of keys underneath it: + // If we get this far, it's a match! Whatever happens next should consume + // the tuple of keys and the subject: pc->on_top -= 2; - ADDOP(c, POP_TOP); if (star_target) { - // If we have a starred name, bind a dict of remaining items to it: - ADDOP(c, COPY_DICT_WITHOUT_KEYS); + // If we have a starred name, bind a dict of remaining items to it (this may + // seem a bit inefficient, but keys is rarely big enough to actually impact + // runtime): + // rest = dict(TOS1) + // for key in TOS: + // del rest[key] + ADDOP_I(c, BUILD_MAP, 0); // [subject, keys, empty] + ADDOP(c, ROT_THREE); // [empty, subject, keys] + ADDOP(c, ROT_TWO); // [empty, keys, subject] + ADDOP_I(c, DICT_UPDATE, 2); // [copy, keys] + ADDOP_I(c, UNPACK_SEQUENCE, size); // [copy, keys...] + while (size) { + ADDOP_I(c, COPY, 1 + size--); // [copy, keys..., copy] + ADDOP(c, ROT_TWO); // [copy, keys..., copy, key] + ADDOP(c, DELETE_SUBSCR); // [copy, keys...] + } RETURN_IF_FALSE(pattern_helper_store_name(c, star_target, pc)); } else { - // Otherwise, we don't care about this tuple of keys anymore: - ADDOP(c, POP_TOP); + ADDOP(c, POP_TOP); // Tuple of keys. + ADDOP(c, POP_TOP); // Subject. } - // Pop the subject: - pc->on_top--; - ADDOP(c, POP_TOP); return 1; error: @@ -8362,6 +8372,11 @@ optimize_basic_block(struct compiler *c, basicblock *bb, PyObject *consts) fold_rotations(inst - oparg + 1, oparg); } break; + case COPY: + if (oparg == 1) { + inst->i_opcode = DUP_TOP; + } + break; default: /* All HAS_CONST opcodes should be handled with LOAD_CONST */ assert (!HAS_CONST(inst->i_opcode)); diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 5c7d3ad544e56..93a9e4ab42cfb 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -33,11 +33,10 @@ static void *opcode_targets[256] = { &&TARGET_MATCH_MAPPING, &&TARGET_MATCH_SEQUENCE, &&TARGET_MATCH_KEYS, - &&TARGET_COPY_DICT_WITHOUT_KEYS, - &&TARGET_PUSH_EXC_INFO, &&TARGET_BINARY_MULTIPLY_INT, - &&TARGET_POP_EXCEPT_AND_RERAISE, + &&TARGET_PUSH_EXC_INFO, &&TARGET_BINARY_MULTIPLY_FLOAT, + &&TARGET_POP_EXCEPT_AND_RERAISE, &&TARGET_BINARY_SUBSCR_ADAPTIVE, &&TARGET_BINARY_SUBSCR_LIST_INT, &&TARGET_BINARY_SUBSCR_TUPLE_INT, @@ -48,6 +47,7 @@ static void *opcode_targets[256] = { &&TARGET_CALL_FUNCTION_LEN, &&TARGET_CALL_FUNCTION_ISINSTANCE, &&TARGET_CALL_FUNCTION_PY_SIMPLE, + &&TARGET_JUMP_ABSOLUTE_QUICK, &&TARGET_WITH_EXCEPT_START, &&TARGET_GET_AITER, &&TARGET_GET_ANEXT, @@ -57,7 +57,7 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_ADD, &&TARGET_INPLACE_SUBTRACT, &&TARGET_INPLACE_MULTIPLY, - &&TARGET_JUMP_ABSOLUTE_QUICK, + &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_INPLACE_MODULO, &&TARGET_STORE_SUBSCR, &&TARGET_DELETE_SUBSCR, @@ -79,15 +79,15 @@ static void *opcode_targets[256] = { &&TARGET_INPLACE_AND, &&TARGET_INPLACE_XOR, &&TARGET_INPLACE_OR, - &&TARGET_LOAD_ATTR_ADAPTIVE, &&TARGET_LOAD_ATTR_INSTANCE_VALUE, + &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, &&TARGET_SETUP_ANNOTATIONS, &&TARGET_YIELD_VALUE, - &&TARGET_LOAD_ATTR_WITH_HINT, &&TARGET_LOAD_ATTR_SLOT, + &&TARGET_LOAD_ATTR_MODULE, &&TARGET_POP_EXCEPT, &&TARGET_STORE_NAME, &&TARGET_DELETE_NAME, @@ -119,7 +119,7 @@ static void *opcode_targets[256] = { &&TARGET_IS_OP, &&TARGET_CONTAINS_OP, &&TARGET_RERAISE, - &&TARGET_LOAD_ATTR_MODULE, + &&TARGET_COPY, &&TARGET_JUMP_IF_NOT_EXC_MATCH, &&TARGET_LOAD_GLOBAL_ADAPTIVE, &&TARGET_LOAD_GLOBAL_MODULE, From webhook-mailer at python.org Wed Oct 27 07:12:26 2021 From: webhook-mailer at python.org (encukou) Date: Wed, 27 Oct 2021 11:12:26 -0000 Subject: [Python-checkins] bpo-42064: Convert `sqlite3` global state to module state (GH-29073) Message-ID: https://github.com/python/cpython/commit/8f24b7dbcbd83311dad510863d8cb41f0e91b464 commit: 8f24b7dbcbd83311dad510863d8cb41f0e91b464 branch: main author: Erlend Egeberg Aasland committer: encukou date: 2021-10-27T13:12:21+02:00 summary: bpo-42064: Convert `sqlite3` global state to module state (GH-29073) files: M Modules/_sqlite/clinic/cursor.c.h M Modules/_sqlite/connection.c M Modules/_sqlite/cursor.c M Modules/_sqlite/microprotocols.c M Modules/_sqlite/microprotocols.h M Modules/_sqlite/module.c M Modules/_sqlite/module.h M Modules/_sqlite/row.c diff --git a/Modules/_sqlite/clinic/cursor.c.h b/Modules/_sqlite/clinic/cursor.c.h index eb3e7ce31ada3..d8a36ac38aaf0 100644 --- a/Modules/_sqlite/clinic/cursor.c.h +++ b/Modules/_sqlite/clinic/cursor.c.h @@ -279,25 +279,14 @@ PyDoc_STRVAR(pysqlite_cursor_close__doc__, "Closes the cursor."); #define PYSQLITE_CURSOR_CLOSE_METHODDEF \ - {"close", (PyCFunction)(void(*)(void))pysqlite_cursor_close, METH_METHOD|METH_FASTCALL|METH_KEYWORDS, pysqlite_cursor_close__doc__}, + {"close", (PyCFunction)pysqlite_cursor_close, METH_NOARGS, pysqlite_cursor_close__doc__}, static PyObject * -pysqlite_cursor_close_impl(pysqlite_Cursor *self, PyTypeObject *cls); +pysqlite_cursor_close_impl(pysqlite_Cursor *self); static PyObject * -pysqlite_cursor_close(pysqlite_Cursor *self, PyTypeObject *cls, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) +pysqlite_cursor_close(pysqlite_Cursor *self, PyObject *Py_UNUSED(ignored)) { - PyObject *return_value = NULL; - static const char * const _keywords[] = { NULL}; - static _PyArg_Parser _parser = {":close", _keywords, 0}; - - if (!_PyArg_ParseStackAndKeywords(args, nargs, kwnames, &_parser - )) { - goto exit; - } - return_value = pysqlite_cursor_close_impl(self, cls); - -exit: - return return_value; + return pysqlite_cursor_close_impl(self); } -/*[clinic end generated code: output=3b5328c1619b7626 input=a9049054013a1b77]*/ +/*[clinic end generated code: output=514f6eb4e4974671 input=a9049054013a1b77]*/ diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index e94c4cbb4e8c3..da2f12e8f99c2 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -59,7 +59,7 @@ clinic_fsconverter(PyObject *pathlike, const char **result) return 0; } -#define clinic_state() (pysqlite_get_state(NULL)) +#define clinic_state() (pysqlite_get_state_by_type(Py_TYPE(self))) #include "clinic/connection.c.h" #undef clinic_state @@ -416,7 +416,8 @@ pysqlite_connection_close_impl(pysqlite_Connection *self) } if (!self->initialized) { - pysqlite_state *state = pysqlite_get_state(NULL); + PyTypeObject *tp = Py_TYPE(self); + pysqlite_state *state = pysqlite_get_state_by_type(tp); PyErr_SetString(state->ProgrammingError, "Base Connection.__init__ not called."); return NULL; diff --git a/Modules/_sqlite/cursor.c b/Modules/_sqlite/cursor.c index ca74a68de4dba..1d7c0b46a616d 100644 --- a/Modules/_sqlite/cursor.c +++ b/Modules/_sqlite/cursor.c @@ -25,7 +25,7 @@ #include "module.h" #include "util.h" -#define clinic_state() (pysqlite_get_state(NULL)) +#define clinic_state() (pysqlite_get_state_by_type(Py_TYPE(self))) #include "clinic/cursor.c.h" #undef clinic_state @@ -966,17 +966,16 @@ pysqlite_cursor_setoutputsize_impl(pysqlite_Cursor *self, PyObject *size, /*[clinic input] _sqlite3.Cursor.close as pysqlite_cursor_close - cls: defining_class - Closes the cursor. [clinic start generated code]*/ static PyObject * -pysqlite_cursor_close_impl(pysqlite_Cursor *self, PyTypeObject *cls) -/*[clinic end generated code: output=a08ab3d772f45438 input=28ba9b532ab46ba0]*/ +pysqlite_cursor_close_impl(pysqlite_Cursor *self) +/*[clinic end generated code: output=b6055e4ec6fe63b6 input=08b36552dbb9a986]*/ { if (!self->connection) { - pysqlite_state *state = pysqlite_get_state_by_cls(cls); + PyTypeObject *tp = Py_TYPE(self); + pysqlite_state *state = pysqlite_get_state_by_type(tp); PyErr_SetString(state->ProgrammingError, "Base Cursor.__init__ not called."); return NULL; diff --git a/Modules/_sqlite/microprotocols.c b/Modules/_sqlite/microprotocols.c index 68e4f7fb166db..95c799d306f30 100644 --- a/Modules/_sqlite/microprotocols.c +++ b/Modules/_sqlite/microprotocols.c @@ -49,7 +49,8 @@ pysqlite_microprotocols_init(PyObject *module) /* pysqlite_microprotocols_add - add a reverse type-caster to the dictionary */ int -pysqlite_microprotocols_add(PyTypeObject *type, PyObject *proto, PyObject *cast) +pysqlite_microprotocols_add(pysqlite_state *state, PyTypeObject *type, + PyObject *proto, PyObject *cast) { PyObject* key; int rc; @@ -61,7 +62,6 @@ pysqlite_microprotocols_add(PyTypeObject *type, PyObject *proto, PyObject *cast) return -1; } - pysqlite_state *state = pysqlite_get_state(NULL); rc = PyDict_SetItem(state->psyco_adapters, key, cast); Py_DECREF(key); diff --git a/Modules/_sqlite/microprotocols.h b/Modules/_sqlite/microprotocols.h index d12bc448596c4..6bde9d01f4529 100644 --- a/Modules/_sqlite/microprotocols.h +++ b/Modules/_sqlite/microprotocols.h @@ -33,8 +33,9 @@ /* used by module.c to init the microprotocols system */ extern int pysqlite_microprotocols_init(PyObject *module); -extern int pysqlite_microprotocols_add( - PyTypeObject *type, PyObject *proto, PyObject *cast); +extern int pysqlite_microprotocols_add(pysqlite_state *state, + PyTypeObject *type, PyObject *proto, + PyObject *cast); extern PyObject *pysqlite_microprotocols_adapt(pysqlite_state *state, PyObject *obj, PyObject *proto, PyObject *alt); diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index 47b1f7a9d0720..e41ac0fc1ab73 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -32,7 +32,7 @@ #error "SQLite 3.7.15 or higher required" #endif -#define clinic_state() (pysqlite_get_state(NULL)) +#define clinic_state() (pysqlite_get_state(module)) #include "clinic/module.c.h" #undef clinic_state @@ -41,8 +41,6 @@ module _sqlite3 [clinic start generated code]*/ /*[clinic end generated code: output=da39a3ee5e6b4b0d input=81e330492d57488e]*/ -pysqlite_state pysqlite_global_state; - // NOTE: This must equal sqlite3.Connection.__init__ argument spec! /*[clinic input] _sqlite3.connect as pysqlite_connect @@ -160,7 +158,7 @@ pysqlite_register_adapter_impl(PyObject *module, PyTypeObject *type, pysqlite_state *state = pysqlite_get_state(module); PyObject *protocol = (PyObject *)state->PrepareProtocolType; - rc = pysqlite_microprotocols_add(type, protocol, caster); + rc = pysqlite_microprotocols_add(state, type, protocol, caster); if (rc == -1) { return NULL; } @@ -395,16 +393,11 @@ static int add_integer_constants(PyObject *module) { return ret; } -static struct PyModuleDef _sqlite3module = { - PyModuleDef_HEAD_INIT, - "_sqlite3", - NULL, - -1, - module_methods, - NULL, - NULL, - NULL, - NULL +struct PyModuleDef _sqlite3module = { + .m_base = PyModuleDef_HEAD_INIT, + .m_name = "_sqlite3", + .m_size = sizeof(pysqlite_state), + .m_methods = module_methods, }; #define ADD_TYPE(module, type) \ diff --git a/Modules/_sqlite/module.h b/Modules/_sqlite/module.h index c273c1f9ed9f2..1d319f1ed5541 100644 --- a/Modules/_sqlite/module.h +++ b/Modules/_sqlite/module.h @@ -63,22 +63,20 @@ typedef struct { extern pysqlite_state pysqlite_global_state; static inline pysqlite_state * -pysqlite_get_state(PyObject *Py_UNUSED(module)) +pysqlite_get_state(PyObject *module) { - return &pysqlite_global_state; // Replace with PyModule_GetState + pysqlite_state *state = (pysqlite_state *)PyModule_GetState(module); + assert(state != NULL); + return state; } +extern struct PyModuleDef _sqlite3module; static inline pysqlite_state * -pysqlite_get_state_by_cls(PyTypeObject *Py_UNUSED(cls)) +pysqlite_get_state_by_type(PyTypeObject *tp) { - return &pysqlite_global_state; // Replace with PyType_GetModuleState -} - -static inline pysqlite_state * -pysqlite_get_state_by_type(PyTypeObject *Py_UNUSED(tp)) -{ - // Replace with _PyType_GetModuleByDef & PyModule_GetState - return &pysqlite_global_state; + PyObject *module = _PyType_GetModuleByDef(tp, &_sqlite3module); + assert(module != NULL); + return pysqlite_get_state(module); } extern const char *pysqlite_error_name(int rc); diff --git a/Modules/_sqlite/row.c b/Modules/_sqlite/row.c index b146c9dc5e3bd..1a1943285ce00 100644 --- a/Modules/_sqlite/row.c +++ b/Modules/_sqlite/row.c @@ -24,7 +24,7 @@ #include "row.h" #include "cursor.h" -#define clinic_state() (pysqlite_get_state(NULL)) +#define clinic_state() (pysqlite_get_state_by_type(type)) #include "clinic/row.c.h" #undef clinic_state @@ -219,7 +219,7 @@ static PyObject* pysqlite_row_richcompare(pysqlite_Row *self, PyObject *_other, if (opid != Py_EQ && opid != Py_NE) Py_RETURN_NOTIMPLEMENTED; - pysqlite_state *state = pysqlite_get_state_by_cls(Py_TYPE(self)); + pysqlite_state *state = pysqlite_get_state_by_type(Py_TYPE(self)); if (PyObject_TypeCheck(_other, state->RowType)) { pysqlite_Row *other = (pysqlite_Row *)_other; int eq = PyObject_RichCompareBool(self->description, other->description, Py_EQ); From webhook-mailer at python.org Wed Oct 27 10:21:39 2021 From: webhook-mailer at python.org (ned-deily) Date: Wed, 27 Oct 2021 14:21:39 -0000 Subject: [Python-checkins] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) Message-ID: https://github.com/python/cpython/commit/bcee6aa31550cfecdc3acecbd0e4447bb0051887 commit: bcee6aa31550cfecdc3acecbd0e4447bb0051887 branch: main author: m-aciek committer: ned-deily date: 2021-10-27T10:21:29-04:00 summary: bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) Co-authored-by: Maciej Olko Co-authored-by: Erlend Egeberg Aasland files: M Doc/requirements.txt diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 2b70af3a4fc6b..dd3c8e62237cf 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -4,6 +4,10 @@ # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. sphinx==3.2.1 +# Docutils version is pinned to a version compatible with Sphinx +# version 3.2.1. It can be removed after bumping Sphinx version to at +# least 3.5.4. +docutils==0.17.1 blurb From webhook-mailer at python.org Wed Oct 27 10:26:31 2021 From: webhook-mailer at python.org (markshannon) Date: Wed, 27 Oct 2021 14:26:31 -0000 Subject: [Python-checkins] Don't make a call at the C level when calling bound-methods from Python code. (GH-29238) Message-ID: https://github.com/python/cpython/commit/4776b07d178b2800374f5c15da182f1215756205 commit: 4776b07d178b2800374f5c15da182f1215756205 branch: main author: Mark Shannon committer: markshannon date: 2021-10-27T15:26:22+01:00 summary: Don't make a call at the C level when calling bound-methods from Python code. (GH-29238) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index a0f4c801e8f8c..bd01e8159c8e4 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4607,8 +4607,21 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr kwnames = NULL; postcall_shrink = 1; call_function: - // Check if the call can be inlined or not function = PEEK(oparg + 1); + if (Py_TYPE(function) == &PyMethod_Type) { + PyObject *meth = ((PyMethodObject *)function)->im_func; + PyObject *self = ((PyMethodObject *)function)->im_self; + Py_INCREF(meth); + Py_INCREF(self); + PEEK(oparg + 1) = self; + Py_DECREF(function); + function = meth; + oparg++; + nargs++; + assert(postcall_shrink >= 1); + postcall_shrink--; + } + // Check if the call can be inlined or not if (Py_TYPE(function) == &PyFunction_Type && tstate->interp->eval_frame == NULL) { int code_flags = ((PyCodeObject*)PyFunction_GET_CODE(function))->co_flags; int is_generator = code_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR); From webhook-mailer at python.org Wed Oct 27 10:37:26 2021 From: webhook-mailer at python.org (ned-deily) Date: Wed, 27 Oct 2021 14:37:26 -0000 Subject: [Python-checkins] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29240) Message-ID: https://github.com/python/cpython/commit/2b7b7c7320cd856df5439afc3c984873678c27d8 commit: 2b7b7c7320cd856df5439afc3c984873678c27d8 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ned-deily date: 2021-10-27T10:37:17-04:00 summary: bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29240) Co-authored-by: Maciej Olko Co-authored-by: Erlend Egeberg Aasland (cherry picked from commit bcee6aa31550cfecdc3acecbd0e4447bb0051887) files: M Doc/requirements.txt diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 2b70af3a4fc6b..dd3c8e62237cf 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -4,6 +4,10 @@ # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. sphinx==3.2.1 +# Docutils version is pinned to a version compatible with Sphinx +# version 3.2.1. It can be removed after bumping Sphinx version to at +# least 3.5.4. +docutils==0.17.1 blurb From webhook-mailer at python.org Wed Oct 27 11:04:53 2021 From: webhook-mailer at python.org (ned-deily) Date: Wed, 27 Oct 2021 15:04:53 -0000 Subject: [Python-checkins] [3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) Message-ID: https://github.com/python/cpython/commit/30c1f18ee62cef301e18488b80b4d329290f7b95 commit: 30c1f18ee62cef301e18488b80b4d329290f7b95 branch: 3.9 author: Ned Deily committer: ned-deily date: 2021-10-27T11:04:38-04:00 summary: [3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) Co-authored-by: Maciej Olko Co-authored-by: Erlend Egeberg Aasland files: M .azure-pipelines/docs-steps.yml M Doc/requirements.txt diff --git a/.azure-pipelines/docs-steps.yml b/.azure-pipelines/docs-steps.yml index eaf4ad01927ae..647daff7a033a 100644 --- a/.azure-pipelines/docs-steps.yml +++ b/.azure-pipelines/docs-steps.yml @@ -12,11 +12,12 @@ steps: inputs: versionSpec: '>=3.6' -- script: python -m pip install sphinx==2.2.0 blurb python-docs-theme +- script: python -m pip install -r requirements.txt + workingDirectory: '$(build.sourcesDirectory)/Doc' displayName: 'Install build dependencies' - ${{ if ne(parameters.latex, 'true') }}: - - script: make check suspicious html PYTHON=python + - script: make check html PYTHON=python workingDirectory: '$(build.sourcesDirectory)/Doc' displayName: 'Build documentation' @@ -31,7 +32,7 @@ steps: - ${{ if eq(parameters.upload, 'true') }}: - task: PublishBuildArtifacts at 1 displayName: 'Publish docs' - + inputs: PathToPublish: '$(build.sourcesDirectory)/Doc/build' ArtifactName: docs diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 47b78eeac817e..cb21ed20397b2 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -4,6 +4,10 @@ # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. sphinx==2.4.4 +# Docutils version is pinned to a version compatible with Sphinx +# version 2.4.4. It can be removed after bumping Sphinx version to at +# least 3.5.4. +docutils==0.17.1 blurb From webhook-mailer at python.org Wed Oct 27 15:15:26 2021 From: webhook-mailer at python.org (ambv) Date: Wed, 27 Oct 2021 19:15:26 -0000 Subject: [Python-checkins] bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) Message-ID: https://github.com/python/cpython/commit/35e1ff38ee67ee543d9fcb268c3552c5397f9b3f commit: 35e1ff38ee67ee543d9fcb268c3552c5397f9b3f branch: main author: Sam Gross committer: ambv date: 2021-10-27T21:15:13+02:00 summary: bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) Objects that support garbage collection ("container" objects) should call PyObject_GC_UnTrack() from their destructors before clearing any fields which may point to other "container" objects. files: M Doc/c-api/gcsupport.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes.rst diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index 4bd2fb3837a1a..8c90d1e8991c1 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -33,6 +33,14 @@ Constructors for container types must conform to two rules: #. Once all the fields which may contain references to other containers are initialized, it must call :c:func:`PyObject_GC_Track`. +Similarly, the deallocator for the object must conform to a similar pair of +rules: + +#. Before fields which refer to other containers are invalidated, + :c:func:`PyObject_GC_UnTrack` must be called. + +#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. + .. warning:: If a type adds the Py_TPFLAGS_HAVE_GC, then it *must* implement at least a :c:member:`~PyTypeObject.tp_traverse` handler or explicitly use one @@ -100,14 +108,6 @@ Constructors for container types must conform to two rules: .. versionadded:: 3.9 -Similarly, the deallocator for the object must conform to a similar pair of -rules: - -#. Before fields which refer to other containers are invalidated, - :c:func:`PyObject_GC_UnTrack` must be called. - -#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. - .. c:function:: void PyObject_GC_Del(void *op) diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index d3d23e1cf2ee3..541287956f0a5 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -668,6 +668,18 @@ and :c:type:`PyType_Type` effectively act as defaults.) :c:func:`PyObject_GC_Del` if the instance was allocated using :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. + If the type supports garbage collection (has the :const:`Py_TPFLAGS_HAVE_GC` + flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack` + before clearing any member fields. + + .. code-block:: c + + static void foo_dealloc(foo_object *self) { + PyObject_GC_UnTrack(self); + Py_CLEAR(self->ref); + Py_TYPE(self)->tp_free((PyObject *)self); + } + Finally, if the type is heap allocated (:const:`Py_TPFLAGS_HEAPTYPE`), the deallocator should decrement the reference count for its type object after calling the type deallocator. In order to avoid dangling pointers, the diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index 6e17897ed2c80..23ec8bce8c5ac 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -73,7 +73,19 @@ function:: newdatatype_dealloc(newdatatypeobject *obj) { free(obj->obj_UnderlyingDatatypePtr); - Py_TYPE(obj)->tp_free(obj); + Py_TYPE(obj)->tp_free((PyObject *)obj); + } + +If your type supports garbage collection, the destructor should call +:c:func:`PyObject_GC_UnTrack` before clearing any member fields:: + + static void + newdatatype_dealloc(newdatatypeobject *obj) + { + PyObject_GC_UnTrack(obj); + Py_CLEAR(obj->other_obj); + ... + Py_TYPE(obj)->tp_free((PyObject *)obj); } .. index:: From webhook-mailer at python.org Wed Oct 27 17:27:43 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Oct 2021 21:27:43 -0000 Subject: [Python-checkins] bpo-45562: Print tokenizer debug messages to stderr (GH-29250) Message-ID: https://github.com/python/cpython/commit/10bbd41ba8c88bc102df108a4e0444abc7c5ea43 commit: 10bbd41ba8c88bc102df108a4e0444abc7c5ea43 branch: main author: Pablo Galindo Salgado committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-27T14:27:34-07:00 summary: bpo-45562: Print tokenizer debug messages to stderr (GH-29250) files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 705da00463d1a..4329cdd226e9c 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1047,9 +1047,9 @@ tok_nextc(struct tok_state *tok) } #if defined(Py_DEBUG) if (Py_DebugFlag) { - printf("line[%d] = ", tok->lineno); + fprintf(stderr, "line[%d] = ", tok->lineno); print_escape(stdout, tok->cur, tok->inp - tok->cur); - printf(" tok->done = %d\n", tok->done); + fprintf(stderr, " tok->done = %d\n", tok->done); } #endif if (!rc) { @@ -2127,8 +2127,8 @@ _PyTokenizer_FindEncodingFilename(int fd, PyObject *filename) void tok_dump(int type, char *start, char *end) { - printf("%s", _PyParser_TokenNames[type]); + fprintf(stderr, "%s", _PyParser_TokenNames[type]); if (type == NAME || type == NUMBER || type == STRING || type == OP) - printf("(%.*s)", (int)(end - start), start); + fprintf(stderr, "(%.*s)", (int)(end - start), start); } #endif // Py_DEBUG From webhook-mailer at python.org Wed Oct 27 17:36:51 2021 From: webhook-mailer at python.org (gvanrossum) Date: Wed, 27 Oct 2021 21:36:51 -0000 Subject: [Python-checkins] bpo-45438: format of inspect.Signature with generic builtins (#29212) Message-ID: https://github.com/python/cpython/commit/d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899 commit: d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899 branch: main author: Martin Rueckl committer: gvanrossum date: 2021-10-27T14:36:41-07:00 summary: bpo-45438: format of inspect.Signature with generic builtins (#29212) Use types.GenericAlias in inspect.formatannotation to correctly add type arguments of builtin types to the string representation of Signatures. Co-authored-by: Martin R?ckl files: A Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst M Lib/inspect.py M Lib/test/test_inspect.py diff --git a/Lib/inspect.py b/Lib/inspect.py index a956acf310f85..14a42fda56c65 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1325,6 +1325,8 @@ def getargvalues(frame): def formatannotation(annotation, base_module=None): if getattr(annotation, '__module__', None) == 'typing': return repr(annotation).replace('typing.', '') + if isinstance(annotation, types.GenericAlias): + return str(annotation) if isinstance(annotation, type): if annotation.__module__ in ('builtins', base_module): return annotation.__qualname__ diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 516efac45e523..7479eb326e6f5 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -3236,6 +3236,17 @@ def foo(): pass self.assertEqual(str(inspect.signature(foo)), '()') + def foo(a: list[str]) -> tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> tuple[str, float]') + + from typing import Tuple + def foo(a: list[str]) -> Tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> Tuple[str, float]') + def test_signature_str_positional_only(self): P = inspect.Parameter S = inspect.Signature diff --git a/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst new file mode 100644 index 0000000000000..cd6cfc1fcd407 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst @@ -0,0 +1 @@ +Fix typing.Signature string representation for generic builtin types. From webhook-mailer at python.org Wed Oct 27 17:45:52 2021 From: webhook-mailer at python.org (pablogsal) Date: Wed, 27 Oct 2021 21:45:52 -0000 Subject: [Python-checkins] bpo-45562: Print tokenizer debug messages to stderr (GH-29250) (GH-29252) Message-ID: https://github.com/python/cpython/commit/038f45230892235e806ef227dfd9484b95687823 commit: 038f45230892235e806ef227dfd9484b95687823 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: pablogsal date: 2021-10-27T22:45:43+01:00 summary: bpo-45562: Print tokenizer debug messages to stderr (GH-29250) (GH-29252) (cherry picked from commit 10bbd41ba8c88bc102df108a4e0444abc7c5ea43) Co-authored-by: Pablo Galindo Salgado Co-authored-by: Pablo Galindo Salgado files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index cd348487e0fd5..6563cdfd7cc60 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1046,9 +1046,9 @@ tok_nextc(struct tok_state *tok) } #if defined(Py_DEBUG) if (Py_DebugFlag) { - printf("line[%d] = ", tok->lineno); + fprintf(stderr, "line[%d] = ", tok->lineno); print_escape(stdout, tok->cur, tok->inp - tok->cur); - printf(" tok->done = %d\n", tok->done); + fprintf(stderr, " tok->done = %d\n", tok->done); } #endif if (!rc) { @@ -2132,9 +2132,9 @@ PyTokenizer_FindEncoding(int fd) void tok_dump(int type, char *start, char *end) { - printf("%s", _PyParser_TokenNames[type]); + fprintf(stderr, "%s", _PyParser_TokenNames[type]); if (type == NAME || type == NUMBER || type == STRING || type == OP) - printf("(%.*s)", (int)(end - start), start); + fprintf(stderr, "(%.*s)", (int)(end - start), start); } #endif From webhook-mailer at python.org Wed Oct 27 17:57:11 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Oct 2021 21:57:11 -0000 Subject: [Python-checkins] bpo-45438: format of inspect.Signature with generic builtins (GH-29212) Message-ID: https://github.com/python/cpython/commit/ce7a6afb797d2ffde45e9e902516b8437c8f9e31 commit: ce7a6afb797d2ffde45e9e902516b8437c8f9e31 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-27T14:57:07-07:00 summary: bpo-45438: format of inspect.Signature with generic builtins (GH-29212) Use types.GenericAlias in inspect.formatannotation to correctly add type arguments of builtin types to the string representation of Signatures. Co-authored-by: Martin R?ckl (cherry picked from commit d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899) Co-authored-by: Martin Rueckl files: A Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst M Lib/inspect.py M Lib/test/test_inspect.py diff --git a/Lib/inspect.py b/Lib/inspect.py index 2374e59f0b402..531b891283b53 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1357,6 +1357,8 @@ def getargvalues(frame): def formatannotation(annotation, base_module=None): if getattr(annotation, '__module__', None) == 'typing': return repr(annotation).replace('typing.', '') + if isinstance(annotation, types.GenericAlias): + return str(annotation) if isinstance(annotation, type): if annotation.__module__ in ('builtins', base_module): return annotation.__qualname__ diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index f0dc6bb45c5a5..4164634417b2a 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -3316,6 +3316,17 @@ def foo(): pass self.assertEqual(str(inspect.signature(foo)), '()') + def foo(a: list[str]) -> tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> tuple[str, float]') + + from typing import Tuple + def foo(a: list[str]) -> Tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> Tuple[str, float]') + def test_signature_str_positional_only(self): P = inspect.Parameter S = inspect.Signature diff --git a/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst new file mode 100644 index 0000000000000..cd6cfc1fcd407 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst @@ -0,0 +1 @@ +Fix typing.Signature string representation for generic builtin types. From webhook-mailer at python.org Wed Oct 27 18:00:28 2021 From: webhook-mailer at python.org (miss-islington) Date: Wed, 27 Oct 2021 22:00:28 -0000 Subject: [Python-checkins] bpo-45438: format of inspect.Signature with generic builtins (GH-29212) Message-ID: https://github.com/python/cpython/commit/21150c6fa330f80747d698e4b883c7b4801a25bd commit: 21150c6fa330f80747d698e4b883c7b4801a25bd branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-27T15:00:18-07:00 summary: bpo-45438: format of inspect.Signature with generic builtins (GH-29212) Use types.GenericAlias in inspect.formatannotation to correctly add type arguments of builtin types to the string representation of Signatures. Co-authored-by: Martin R?ckl (cherry picked from commit d02ffd1b5c0fd8dec6dd2f7e3f2b0cfae48b7899) Co-authored-by: Martin Rueckl files: A Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst M Lib/inspect.py M Lib/test/test_inspect.py diff --git a/Lib/inspect.py b/Lib/inspect.py index 976cbb1a234e4..64605c3cd3f6f 100644 --- a/Lib/inspect.py +++ b/Lib/inspect.py @@ -1233,6 +1233,8 @@ def getargvalues(frame): def formatannotation(annotation, base_module=None): if getattr(annotation, '__module__', None) == 'typing': return repr(annotation).replace('typing.', '') + if isinstance(annotation, types.GenericAlias): + return str(annotation) if isinstance(annotation, type): if annotation.__module__ in ('builtins', base_module): return annotation.__qualname__ diff --git a/Lib/test/test_inspect.py b/Lib/test/test_inspect.py index 17c5d87f8ab45..8a8844e912ad2 100644 --- a/Lib/test/test_inspect.py +++ b/Lib/test/test_inspect.py @@ -3200,6 +3200,17 @@ def foo(): pass self.assertEqual(str(inspect.signature(foo)), '()') + def foo(a: list[str]) -> tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> tuple[str, float]') + + from typing import Tuple + def foo(a: list[str]) -> Tuple[str, float]: + pass + self.assertEqual(str(inspect.signature(foo)), + '(a: list[str]) -> Tuple[str, float]') + def test_signature_str_positional_only(self): P = inspect.Parameter S = inspect.Signature diff --git a/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst new file mode 100644 index 0000000000000..cd6cfc1fcd407 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-27-10-05-39.bpo-45438.Xz5lGU.rst @@ -0,0 +1 @@ +Fix typing.Signature string representation for generic builtin types. From webhook-mailer at python.org Thu Oct 28 03:48:29 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Thu, 28 Oct 2021 07:48:29 -0000 Subject: [Python-checkins] bpo-44904: Fix classmethod property bug in doctest module (GH-28838) Message-ID: https://github.com/python/cpython/commit/b1302abcc8a4be5f39b4d60a1ce28032b77655b3 commit: b1302abcc8a4be5f39b4d60a1ce28032b77655b3 branch: main author: Alex Waygood committer: serhiy-storchaka date: 2021-10-28T10:48:02+03:00 summary: bpo-44904: Fix classmethod property bug in doctest module (GH-28838) The doctest module raised an error if a docstring contained an example that attempted to access a classmethod property. (Stacking '@classmethod' on top of `@property` has been supported since Python 3.9; see https://docs.python.org/3/howto/descriptor.html#class-methods.) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst M Lib/doctest.py M Lib/test/test_doctest.py M Misc/ACKS diff --git a/Lib/doctest.py b/Lib/doctest.py index ba898f65403df..b27cbdfed46ff 100644 --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -1034,10 +1034,8 @@ def _find(self, tests, obj, name, module, source_lines, globs, seen): if inspect.isclass(obj) and self._recurse: for valname, val in obj.__dict__.items(): # Special handling for staticmethod/classmethod. - if isinstance(val, staticmethod): - val = getattr(obj, valname) - if isinstance(val, classmethod): - val = getattr(obj, valname).__func__ + if isinstance(val, (staticmethod, classmethod)): + val = val.__func__ # Recurse to methods, properties, and nested classes. if ((inspect.isroutine(val) or inspect.isclass(val) or diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 8423cafa8c796..407a14c7ddf67 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -96,6 +96,17 @@ def a_classmethod(cls, v): 22 """) + a_class_attribute = 42 + + @classmethod + @property + def a_classmethod_property(cls): + """ + >>> print(SampleClass.a_classmethod_property) + 42 + """ + return cls.a_class_attribute + class NestedClass: """ >>> x = SampleClass.NestedClass(5) @@ -501,6 +512,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -556,6 +568,7 @@ def basics(): r""" 1 some_module.SampleClass.NestedClass.__init__ 1 some_module.SampleClass.__init__ 2 some_module.SampleClass.a_classmethod + 1 some_module.SampleClass.a_classmethod_property 1 some_module.SampleClass.a_property 1 some_module.SampleClass.a_staticmethod 1 some_module.SampleClass.double @@ -597,6 +610,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -617,6 +631,7 @@ def basics(): r""" 0 SampleClass.NestedClass.square 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double diff --git a/Misc/ACKS b/Misc/ACKS index 23c92abb4d02a..204293fa50d9c 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1885,6 +1885,7 @@ Bob Watson Colin Watson David Watson Aaron Watters +Alex Waygood Henrik Weber Leon Weber Steve Weber diff --git a/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst new file mode 100644 index 0000000000000..b02d499d23500 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst @@ -0,0 +1,3 @@ +Fix bug in the :mod:`doctest` module that caused it to fail if a docstring +included an example with a ``classmethod`` ``property``. Patch by Alex +Waygood. From webhook-mailer at python.org Thu Oct 28 04:09:57 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Oct 2021 08:09:57 -0000 Subject: [Python-checkins] bpo-44904: Fix classmethod property bug in doctest module (GH-28838) Message-ID: https://github.com/python/cpython/commit/1f45cc0dfa9a8febfc256411c803b4536719db97 commit: 1f45cc0dfa9a8febfc256411c803b4536719db97 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-28T01:09:41-07:00 summary: bpo-44904: Fix classmethod property bug in doctest module (GH-28838) The doctest module raised an error if a docstring contained an example that attempted to access a classmethod property. (Stacking '@classmethod' on top of `@property` has been supported since Python 3.9; see https://docs.python.org/3/howto/descriptor.htmlGH-class-methods.) Co-authored-by: Serhiy Storchaka (cherry picked from commit b1302abcc8a4be5f39b4d60a1ce28032b77655b3) Co-authored-by: Alex Waygood files: A Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst M Lib/doctest.py M Lib/test/test_doctest.py M Misc/ACKS diff --git a/Lib/doctest.py b/Lib/doctest.py index ba898f65403df..b27cbdfed46ff 100644 --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -1034,10 +1034,8 @@ def _find(self, tests, obj, name, module, source_lines, globs, seen): if inspect.isclass(obj) and self._recurse: for valname, val in obj.__dict__.items(): # Special handling for staticmethod/classmethod. - if isinstance(val, staticmethod): - val = getattr(obj, valname) - if isinstance(val, classmethod): - val = getattr(obj, valname).__func__ + if isinstance(val, (staticmethod, classmethod)): + val = val.__func__ # Recurse to methods, properties, and nested classes. if ((inspect.isroutine(val) or inspect.isclass(val) or diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index 7f8ccd3896ead..9703f871ad640 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -96,6 +96,17 @@ def a_classmethod(cls, v): 22 """) + a_class_attribute = 42 + + @classmethod + @property + def a_classmethod_property(cls): + """ + >>> print(SampleClass.a_classmethod_property) + 42 + """ + return cls.a_class_attribute + class NestedClass: """ >>> x = SampleClass.NestedClass(5) @@ -501,6 +512,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -556,6 +568,7 @@ def basics(): r""" 1 some_module.SampleClass.NestedClass.__init__ 1 some_module.SampleClass.__init__ 2 some_module.SampleClass.a_classmethod + 1 some_module.SampleClass.a_classmethod_property 1 some_module.SampleClass.a_property 1 some_module.SampleClass.a_staticmethod 1 some_module.SampleClass.double @@ -597,6 +610,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -617,6 +631,7 @@ def basics(): r""" 0 SampleClass.NestedClass.square 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double diff --git a/Misc/ACKS b/Misc/ACKS index 4590145a87abe..94b0ed0b241cd 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1880,6 +1880,7 @@ Bob Watson Colin Watson David Watson Aaron Watters +Alex Waygood Henrik Weber Leon Weber Steve Weber diff --git a/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst new file mode 100644 index 0000000000000..b02d499d23500 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst @@ -0,0 +1,3 @@ +Fix bug in the :mod:`doctest` module that caused it to fail if a docstring +included an example with a ``classmethod`` ``property``. Patch by Alex +Waygood. From webhook-mailer at python.org Thu Oct 28 04:13:54 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Oct 2021 08:13:54 -0000 Subject: [Python-checkins] bpo-44904: Fix classmethod property bug in doctest module (GH-28838) Message-ID: https://github.com/python/cpython/commit/8365a5b5abe51cbe4151d89a5d0a993273320067 commit: 8365a5b5abe51cbe4151d89a5d0a993273320067 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-28T01:13:45-07:00 summary: bpo-44904: Fix classmethod property bug in doctest module (GH-28838) The doctest module raised an error if a docstring contained an example that attempted to access a classmethod property. (Stacking '@classmethod' on top of `@property` has been supported since Python 3.9; see https://docs.python.org/3/howto/descriptor.htmlGH-class-methods.) Co-authored-by: Serhiy Storchaka (cherry picked from commit b1302abcc8a4be5f39b4d60a1ce28032b77655b3) Co-authored-by: Alex Waygood files: A Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst M Lib/doctest.py M Lib/test/test_doctest.py M Misc/ACKS diff --git a/Lib/doctest.py b/Lib/doctest.py index baa503c83f875..d2c8828e5ed97 100644 --- a/Lib/doctest.py +++ b/Lib/doctest.py @@ -1022,10 +1022,8 @@ def _find(self, tests, obj, name, module, source_lines, globs, seen): if inspect.isclass(obj) and self._recurse: for valname, val in obj.__dict__.items(): # Special handling for staticmethod/classmethod. - if isinstance(val, staticmethod): - val = getattr(obj, valname) - if isinstance(val, classmethod): - val = getattr(obj, valname).__func__ + if isinstance(val, (staticmethod, classmethod)): + val = val.__func__ # Recurse to methods, properties, and nested classes. if ((inspect.isroutine(val) or inspect.isclass(val) or diff --git a/Lib/test/test_doctest.py b/Lib/test/test_doctest.py index af5513c631777..47b8575cec563 100644 --- a/Lib/test/test_doctest.py +++ b/Lib/test/test_doctest.py @@ -93,6 +93,17 @@ def a_classmethod(cls, v): 22 """) + a_class_attribute = 42 + + @classmethod + @property + def a_classmethod_property(cls): + """ + >>> print(SampleClass.a_classmethod_property) + 42 + """ + return cls.a_class_attribute + class NestedClass: """ >>> x = SampleClass.NestedClass(5) @@ -498,6 +509,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -553,6 +565,7 @@ def basics(): r""" 1 some_module.SampleClass.NestedClass.__init__ 1 some_module.SampleClass.__init__ 2 some_module.SampleClass.a_classmethod + 1 some_module.SampleClass.a_classmethod_property 1 some_module.SampleClass.a_property 1 some_module.SampleClass.a_staticmethod 1 some_module.SampleClass.double @@ -594,6 +607,7 @@ def basics(): r""" 1 SampleClass.NestedClass.__init__ 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double @@ -614,6 +628,7 @@ def basics(): r""" 0 SampleClass.NestedClass.square 1 SampleClass.__init__ 2 SampleClass.a_classmethod + 1 SampleClass.a_classmethod_property 1 SampleClass.a_property 1 SampleClass.a_staticmethod 1 SampleClass.double diff --git a/Misc/ACKS b/Misc/ACKS index 59ae1462b86a2..ac893acbf3e46 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1846,6 +1846,7 @@ Bob Watson Colin Watson David Watson Aaron Watters +Alex Waygood Henrik Weber Leon Weber Steve Weber diff --git a/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst new file mode 100644 index 0000000000000..b02d499d23500 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-09-18-42-27.bpo-44904.RlW5h8.rst @@ -0,0 +1,3 @@ +Fix bug in the :mod:`doctest` module that caused it to fail if a docstring +included an example with a ``classmethod`` ``property``. Patch by Alex +Waygood. From webhook-mailer at python.org Thu Oct 28 05:57:43 2021 From: webhook-mailer at python.org (tiran) Date: Thu, 28 Oct 2021 09:57:43 -0000 Subject: [Python-checkins] bpo-45548: Fix out-of-tree and Debian builds (GH-29263) Message-ID: https://github.com/python/cpython/commit/4c95fb4640c0247903562dae4478158b348cea6d commit: 4c95fb4640c0247903562dae4478158b348cea6d branch: main author: Christian Heimes committer: tiran date: 2021-10-28T11:57:38+02:00 summary: bpo-45548: Fix out-of-tree and Debian builds (GH-29263) Add Modules subdirs to SRCDIRS to generate directories for out-of-tree object files. Debian wants ncurses lib. Works on Fedora, too. Debian also needs pkg-config to detect correct flags. Remove more outdated comments. Makefile now tracks header dependencies -lintl is injected by configure when needed. Build _dbm with gdbm-compat. Group some modules by purpose. socket, select, and mmap work on Windows, too. files: M .github/workflows/posix-deps-apt.sh M Modules/Setup M configure M configure.ac diff --git a/.github/workflows/posix-deps-apt.sh b/.github/workflows/posix-deps-apt.sh index 0119843e47eeb..fc4aaba552ad2 100755 --- a/.github/workflows/posix-deps-apt.sh +++ b/.github/workflows/posix-deps-apt.sh @@ -3,6 +3,7 @@ apt-get update apt-get -yq install \ build-essential \ + pkg-config \ ccache \ gdb \ lcov \ diff --git a/Modules/Setup b/Modules/Setup index 992ffecb334d5..b4eae38fdff60 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -38,7 +38,9 @@ # # = # -# which defines a Make variable definition inserted into Makefile.in +# which defines a Make variable definition inserted into Makefile.in. +# You can also use any Make variable that is detected by configure and +# defined in Makefile.pre.in, e.g. OpenSSL flags $(OPENSSL_INCLUDES). # # The build process works like this: # @@ -65,6 +67,11 @@ # platform should be listed below. The distribution comes with all # modules enabled that are supported by most platforms and don't # require you to download sources from elsewhere. +# +# NOTE: Avoid editing this file directly. Local changes should go into +# Modules/Setup.local file. To enable all modules for testing, run +# +# sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.local # Some special rules to define PYTHONPATH. @@ -103,12 +110,12 @@ _abc _abc.c _codecs _codecsmodule.c _functools _functoolsmodule.c _io -I$(srcdir)/Modules/_io _io/_iomodule.c _io/iobase.c _io/fileio.c _io/bytesio.c _io/bufferedio.c _io/textio.c _io/stringio.c -_locale _localemodule.c # -lintl +_locale _localemodule.c _operator _operator.c _signal signalmodule.c _sre _sre.c _stat _stat.c -_symtable symtablemodule.c # setup.py can't track the .h file that _symtable depends on. +_symtable symtablemodule.c _thread _threadmodule.c _tracemalloc _tracemalloc.c # See bpo-35053 as to why this is built in. _weakref _weakref.c @@ -129,47 +136,30 @@ time timemodule.c # Python binary, or need to specify some odd set of compiler switches, # you can uncomment the appropriate lines below. -# To enable all modules for testing, run -# sed -n -E 's/^#([a-z_\*].*)$/\1/p' Modules/Setup > Modules/Setup.local - # Uncommenting the following line tells makesetup that all following # modules are to be built as shared libraries (see above for more # detail; also note that *static* or *disabled* cancels this effect): #*shared* -# Modules that should always be present (non UNIX dependent): +# Modules that should always be present (POSIX and Windows): #_asyncio _asynciomodule.c #_bisect _bisectmodule.c -#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c -#_codecs_cn cjkcodecs/_codecs_cn.c -#_codecs_hk cjkcodecs/_codecs_hk.c -#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c -#_codecs_jp cjkcodecs/_codecs_jp.c -#_codecs_kr cjkcodecs/_codecs_kr.c -#_codecs_tw cjkcodecs/_codecs_tw.c #_contextvars _contextvarsmodule.c #_csv _csv.c #_datetime _datetimemodule.c # UNIVERSAL: let mpdecimal.h detect settings #_decimal -DUNIVERSAL -I$(srcdir)/Modules/_decimal/libmpdec _decimal/_decimal.c _decimal/libmpdec/basearith.c _decimal/libmpdec/constants.c _decimal/libmpdec/context.c _decimal/libmpdec/convolute.c _decimal/libmpdec/crt.c _decimal/libmpdec/difradix2.c _decimal/libmpdec/fnt.c _decimal/libmpdec/fourstep.c _decimal/libmpdec/io.c _decimal/libmpdec/mpalloc.c _decimal/libmpdec/mpdecimal.c _decimal/libmpdec/numbertheory.c _decimal/libmpdec/sixstep.c _decimal/libmpdec/transpose.c -#_elementtree -I$(srcdir)/Modules/expat _elementtree.c #_heapq _heapqmodule.c #_json _json.c #_lsprof _lsprof.c rotatingtree.c -#_md5 md5module.c -#_multibytecodec cjkcodecs/multibytecodec.c #_multiprocessing -I$(srcdir)/Modules/_multiprocessing _multiprocessing/multiprocessing.c _multiprocessing/semaphore.c #_opcode _opcode.c #_pickle _pickle.c -#_posixsubprocess _posixsubprocess.c #_queue _queuemodule.c #_random _randommodule.c -#_sha1 sha1module.c -#_sha256 sha256module.c -#_sha512 sha512module.c -#_sha3 _sha3/sha3module.c +#_socket socketmodule.c #_statistics _statisticsmodule.c #_struct _struct.c #_typing _typingmodule.c @@ -179,53 +169,65 @@ time timemodule.c #binascii binascii.c #cmath cmathmodule.c #math mathmodule.c +#mmap mmapmodule.c +#select selectmodule.c + +# XML +#_elementtree -I$(srcdir)/Modules/expat _elementtree.c #pyexpat -I$(srcdir)/Modules/expat expat/xmlparse.c expat/xmlrole.c expat/xmltok.c pyexpat.c -#unicodedata unicodedata.c -# Modules with some UNIX dependencies -- on by default: -# (If you have a really backward UNIX, select and socket may not be -# supported...) +# hashing builtins +#_blake2 _blake2/blake2module.c _blake2/blake2b_impl.c _blake2/blake2s_impl.c +#_md5 md5module.c +#_sha1 sha1module.c +#_sha256 sha256module.c +#_sha512 sha512module.c +#_sha3 _sha3/sha3module.c +# text encodings and unicode +#_codecs_cn cjkcodecs/_codecs_cn.c +#_codecs_hk cjkcodecs/_codecs_hk.c +#_codecs_iso2022 cjkcodecs/_codecs_iso2022.c +#_codecs_jp cjkcodecs/_codecs_jp.c +#_codecs_kr cjkcodecs/_codecs_kr.c +#_codecs_tw cjkcodecs/_codecs_tw.c +#_multibytecodec cjkcodecs/multibytecodec.c +#unicodedata unicodedata.c + +# Modules with some UNIX dependencies + +#_posixsubprocess _posixsubprocess.c #_posixshmem -I$(srcdir)/Modules/_multiprocessing _multiprocessing/posixshmem.c -lrt -#_socket socketmodule.c # socket(2) -#fcntl fcntlmodule.c # fcntl(2) and ioctl(2) -#grp grpmodule.c # grp(3) -#mmap mmapmodule.c # Also works on win32. +#fcntl fcntlmodule.c +#grp grpmodule.c #ossaudiodev ossaudiodev.c -#select selectmodule.c # select(2); not on ancient System V -#spwd spwdmodule.c # spwd(3) +#resource resource.c +#spwd spwdmodule.c #syslog syslogmodule.c +#termios termios.c -# Some more UNIX dependent modules -- off by default, since these -# are not supported by all UNIX systems: +# Modules with UNIX dependencies that require external libraries -#_crypt _cryptmodule.c -lcrypt # crypt(3); breaks many builds. -#nis nismodule.c -I/usr/include/tirpc -lnsl -ltirpc # Sun yellow pages -- not everywhere -#termios termios.c # Steen Lumholt's termios module -#resource resource.c # Jeremy Hylton's rlimit interface +#_crypt _cryptmodule.c -lcrypt +#nis nismodule.c -I/usr/include/tirpc -lnsl -ltirpc # Modules that require external libraries. #_bz2 _bz2module.c -lbz2 #_ctypes _ctypes/_ctypes.c _ctypes/callbacks.c _ctypes/callproc.c _ctypes/stgdict.c _ctypes/cfield.c -ldl -lffi -DHAVE_FFI_PREP_CIF_VAR -DHAVE_FFI_PREP_CLOSURE_LOC -DHAVE_FFI_CLOSURE_ALLOC -# _dbm _dbmmodule.c # -lndbm # dbm(3) +# The _dbm module supports NDBM, GDBM with compat module, and Berkeley DB. +#_dbm _dbmmodule.c -lgdbm_compat -DHAVE_NDBM_H #_gdbm _gdbmmodule.c -lgdbm #_lzma _lzmamodule.c -llzma #_sqlite3 _sqlite/connection.c _sqlite/cursor.c _sqlite/microprotocols.c _sqlite/module.c _sqlite/prepare_protocol.c _sqlite/row.c _sqlite/statement.c _sqlite/util.c -lsqlite3 #_uuid _uuidmodule.c -luuid #zlib zlibmodule.c -lz -# GNU readline. Unlike previous Python incarnations, GNU readline is -# now incorporated in an optional module, configured in the Setup file -# instead of by a configure script switch. You may have to insert a -# -L option pointing to the directory where libreadline.* lives, -# and you may have to change -ltermcap to -ltermlib or perhaps remove -# it, depending on your system -- see the GNU readline instructions. -# It's okay for this to be a shared library, too. - +# The readline module also supports libeditline (-leditline). +# Some systems may require -ltermcap or -ltermlib. #readline readline.c -lreadline -ltermcap -# To dynamically link OpenSSL: +# OpenSSL bindings #_ssl _ssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) $(OPENSSL_LIBS) #_hashlib _hashopenssl.c $(OPENSSL_INCLUDES) $(OPENSSL_LDFLAGS) -lcrypto @@ -277,13 +279,8 @@ time timemodule.c # *** Always uncomment this; X11 libraries to link with: # -lX11 -# Curses support, requiring the System V version of curses, often -# provided by the ncurses library. e.g. on Linux, link with -lncurses -# instead of -lcurses). - -#_curses -lcurses -lcursesw -ltermcap _cursesmodule.c - -# Wrapper for the panel library that's part of ncurses and SYSV curses. +# Some system have -lcurses +#_curses -lncurses -lncursesw -ltermcap _cursesmodule.c #_curses_panel -lpanel -lncurses _curses_panel.c # macOS specific modules diff --git a/configure b/configure index 72463d2271497..a32c9100d9842 100755 --- a/configure +++ b/configure @@ -17288,7 +17288,23 @@ do done -SRCDIRS="Parser Objects Python Modules Modules/_io Programs" +SRCDIRS="\ + Modules \ + Modules/_blake2 \ + Modules/_ctypes \ + Modules/_decimal \ + Modules/_decimal/libmpdec \ + Modules/_io \ + Modules/_multiprocessing \ + Modules/_sha3 \ + Modules/_sqlite \ + Modules/_xxtestfuzz \ + Modules/cjkcodecs \ + Modules/expat \ + Objects \ + Parser \ + Programs \ + Python" { $as_echo "$as_me:${as_lineno-$LINENO}: checking for build directories" >&5 $as_echo_n "checking for build directories... " >&6; } for dir in $SRCDIRS; do diff --git a/configure.ac b/configure.ac index 5674093fe67a4..ccc5e1f74d83e 100644 --- a/configure.ac +++ b/configure.ac @@ -5636,7 +5636,23 @@ do done AC_SUBST(SRCDIRS) -SRCDIRS="Parser Objects Python Modules Modules/_io Programs" +SRCDIRS="\ + Modules \ + Modules/_blake2 \ + Modules/_ctypes \ + Modules/_decimal \ + Modules/_decimal/libmpdec \ + Modules/_io \ + Modules/_multiprocessing \ + Modules/_sha3 \ + Modules/_sqlite \ + Modules/_xxtestfuzz \ + Modules/cjkcodecs \ + Modules/expat \ + Objects \ + Parser \ + Programs \ + Python" AC_MSG_CHECKING(for build directories) for dir in $SRCDIRS; do if test ! -d $dir; then From webhook-mailer at python.org Thu Oct 28 06:11:26 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 28 Oct 2021 10:11:26 -0000 Subject: [Python-checkins] bpo-30570: Fix segfault on buildbots caused by stack overflow from recursion in tests (GH-29258) Message-ID: https://github.com/python/cpython/commit/d56375a0dd4cee162081b173310298a3d32af293 commit: d56375a0dd4cee162081b173310298a3d32af293 branch: main author: Dennis Sweeney <36520290+sweeneyde at users.noreply.github.com> committer: pablogsal date: 2021-10-28T11:11:18+01:00 summary: bpo-30570: Fix segfault on buildbots caused by stack overflow from recursion in tests (GH-29258) * Don't stackoveflow on debug builds * Also catch the pickletester case files: M Lib/test/pickletester.py M Lib/test/test_isinstance.py diff --git a/Lib/test/pickletester.py b/Lib/test/pickletester.py index 3e2c781caa11a..f13d42f664880 100644 --- a/Lib/test/pickletester.py +++ b/Lib/test/pickletester.py @@ -2383,7 +2383,8 @@ def test_bad_getattr(self): # Issue #3514: crash when there is an infinite loop in __getattr__ x = BadGetattr() for proto in protocols: - self.assertRaises(RuntimeError, self.dumps, x, proto) + with support.infinite_recursion(): + self.assertRaises(RuntimeError, self.dumps, x, proto) def test_reduce_bad_iterator(self): # Issue4176: crash when 4th and 5th items of __reduce__() diff --git a/Lib/test/test_isinstance.py b/Lib/test/test_isinstance.py index 6ab44be9a26ba..9d37cff990338 100644 --- a/Lib/test/test_isinstance.py +++ b/Lib/test/test_isinstance.py @@ -5,6 +5,7 @@ import unittest import sys import typing +from test import support @@ -266,12 +267,14 @@ def test_subclass_tuple(self): def test_subclass_recursion_limit(self): # make sure that issubclass raises RecursionError before the C stack is # blown - self.assertRaises(RecursionError, blowstack, issubclass, str, str) + with support.infinite_recursion(): + self.assertRaises(RecursionError, blowstack, issubclass, str, str) def test_isinstance_recursion_limit(self): # make sure that issubclass raises RecursionError before the C stack is # blown - self.assertRaises(RecursionError, blowstack, isinstance, '', str) + with support.infinite_recursion(): + self.assertRaises(RecursionError, blowstack, isinstance, '', str) def test_subclass_with_union(self): self.assertTrue(issubclass(int, int | float | int)) @@ -308,19 +311,19 @@ class X: @property def __bases__(self): return self.__bases__ - - self.assertRaises(RecursionError, issubclass, X(), int) - self.assertRaises(RecursionError, issubclass, int, X()) - self.assertRaises(RecursionError, isinstance, 1, X()) + with support.infinite_recursion(): + self.assertRaises(RecursionError, issubclass, X(), int) + self.assertRaises(RecursionError, issubclass, int, X()) + self.assertRaises(RecursionError, isinstance, 1, X()) def test_infinite_recursion_via_bases_tuple(self): """Regression test for bpo-30570.""" class Failure(object): def __getattr__(self, attr): return (self, None) - - with self.assertRaises(RecursionError): - issubclass(Failure(), int) + with support.infinite_recursion(): + with self.assertRaises(RecursionError): + issubclass(Failure(), int) def test_infinite_cycle_in_bases(self): """Regression test for bpo-30570.""" @@ -328,7 +331,8 @@ class X: @property def __bases__(self): return (self, self, self) - self.assertRaises(RecursionError, issubclass, X(), int) + with support.infinite_recursion(): + self.assertRaises(RecursionError, issubclass, X(), int) def test_infinitely_many_bases(self): """Regression test for bpo-30570.""" @@ -341,7 +345,8 @@ class B: pass A.__getattr__ = B.__getattr__ = X.__getattr__ return (A(), B()) - self.assertRaises(RecursionError, issubclass, X(), int) + with support.infinite_recursion(): + self.assertRaises(RecursionError, issubclass, X(), int) def blowstack(fxn, arg, compare_to): From webhook-mailer at python.org Thu Oct 28 06:43:13 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Thu, 28 Oct 2021 10:43:13 -0000 Subject: [Python-checkins] Fix typo in Counter documentation (GH-29223) Message-ID: https://github.com/python/cpython/commit/0a68b3603fbc0aaf9eeb8ce8b42b78d6fa7cfa78 commit: 0a68b3603fbc0aaf9eeb8ce8b42b78d6fa7cfa78 branch: main author: Justinas Petuchovas committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-28T18:43:09+08:00 summary: Fix typo in Counter documentation (GH-29223) Co-authored-by: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> files: M Doc/library/collections.rst diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 4ba197e11e97b..8bf3cb6cb12da 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -343,7 +343,7 @@ All of those tests treat missing elements as having zero counts so that ``Counter(a=1) == Counter(a=1, b=0)`` returns true. .. versionadded:: 3.10 - Rich comparison operations we were added + Rich comparison operations were added. .. versionchanged:: 3.10 In equality tests, missing elements are treated as having zero counts. From webhook-mailer at python.org Thu Oct 28 07:06:10 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Oct 2021 11:06:10 -0000 Subject: [Python-checkins] Fix typo in Counter documentation (GH-29223) Message-ID: https://github.com/python/cpython/commit/99495b8afffdc62145598516dbdf99e64b6249bd commit: 99495b8afffdc62145598516dbdf99e64b6249bd branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-28T04:06:04-07:00 summary: Fix typo in Counter documentation (GH-29223) Co-authored-by: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> (cherry picked from commit 0a68b3603fbc0aaf9eeb8ce8b42b78d6fa7cfa78) Co-authored-by: Justinas Petuchovas files: M Doc/library/collections.rst diff --git a/Doc/library/collections.rst b/Doc/library/collections.rst index 4ba197e11e97b..8bf3cb6cb12da 100644 --- a/Doc/library/collections.rst +++ b/Doc/library/collections.rst @@ -343,7 +343,7 @@ All of those tests treat missing elements as having zero counts so that ``Counter(a=1) == Counter(a=1, b=0)`` returns true. .. versionadded:: 3.10 - Rich comparison operations we were added + Rich comparison operations were added. .. versionchanged:: 3.10 In equality tests, missing elements are treated as having zero counts. From webhook-mailer at python.org Thu Oct 28 08:59:16 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 28 Oct 2021 12:59:16 -0000 Subject: [Python-checkins] bpo-45637: Store the frame pointer in the cframe (GH-29267) Message-ID: https://github.com/python/cpython/commit/f291404a802d6a1bc50f817c7a26ff3ac9a199ff commit: f291404a802d6a1bc50f817c7a26ff3ac9a199ff branch: main author: Mark Shannon committer: pablogsal date: 2021-10-28T13:59:11+01:00 summary: bpo-45637: Store the frame pointer in the cframe (GH-29267) * Rename 'frame' to 'current_frame' files: M Include/cpython/pystate.h M Modules/_tracemalloc.c M Modules/_xxsubinterpretersmodule.c M Modules/signalmodule.c M Objects/genobject.c M Python/ceval.c M Python/pylifecycle.c M Python/pystate.c M Python/sysmodule.c M Python/traceback.c diff --git a/Include/cpython/pystate.h b/Include/cpython/pystate.h index 528d2a2998c99..cf69c72028a3a 100644 --- a/Include/cpython/pystate.h +++ b/Include/cpython/pystate.h @@ -46,6 +46,8 @@ typedef struct _cframe { * accessed outside of their lifetime. */ int use_tracing; + /* Pointer to the currently executing frame (it can be NULL) */ + struct _interpreter_frame *current_frame; struct _cframe *previous; } CFrame; @@ -77,8 +79,6 @@ struct _ts { struct _ts *next; PyInterpreterState *interp; - /* Borrowed reference to the current frame (it can be NULL) */ - struct _interpreter_frame *frame; int recursion_depth; int recursion_headroom; /* Allow 50 more calls to handle any errors. */ int stackcheck_counter; diff --git a/Modules/_tracemalloc.c b/Modules/_tracemalloc.c index 09d273ad280bb..68e5f0d52dd36 100644 --- a/Modules/_tracemalloc.c +++ b/Modules/_tracemalloc.c @@ -396,7 +396,7 @@ traceback_get_frames(traceback_t *traceback) return; } - InterpreterFrame *pyframe = tstate->frame; + InterpreterFrame *pyframe = tstate->cframe->current_frame; for (; pyframe != NULL;) { if (traceback->nframe < _Py_tracemalloc_config.max_nframe) { tracemalloc_get_frame(pyframe, &traceback->frames[traceback->nframe]); diff --git a/Modules/_xxsubinterpretersmodule.c b/Modules/_xxsubinterpretersmodule.c index 3f683768fcc28..1507abf311430 100644 --- a/Modules/_xxsubinterpretersmodule.c +++ b/Modules/_xxsubinterpretersmodule.c @@ -1839,7 +1839,7 @@ _is_running(PyInterpreterState *interp) } assert(!PyErr_Occurred()); - InterpreterFrame *frame = tstate->frame; + InterpreterFrame *frame = tstate->cframe->current_frame; if (frame == NULL) { return 0; } diff --git a/Modules/signalmodule.c b/Modules/signalmodule.c index 09f4aed9d5ca9..927880376c323 100644 --- a/Modules/signalmodule.c +++ b/Modules/signalmodule.c @@ -1788,7 +1788,7 @@ _PyErr_CheckSignalsTstate(PyThreadState *tstate) */ _Py_atomic_store(&is_tripped, 0); - InterpreterFrame *frame = tstate->frame; + InterpreterFrame *frame = tstate->cframe->current_frame; signal_state_t *state = &signal_global_state; for (int i = 1; i < NSIG; i++) { if (!_Py_atomic_load_relaxed(&Handlers[i].tripped)) { diff --git a/Objects/genobject.c b/Objects/genobject.c index 8bf55155eab1c..efd255d33f9be 100644 --- a/Objects/genobject.c +++ b/Objects/genobject.c @@ -193,7 +193,7 @@ gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult, Py_INCREF(result); _PyFrame_StackPush(frame, result); - frame->previous = tstate->frame; + frame->previous = tstate->cframe->current_frame; gen->gi_exc_state.previous_item = tstate->exc_info; tstate->exc_info = &gen->gi_exc_state; @@ -207,7 +207,7 @@ gen_send_ex2(PyGenObject *gen, PyObject *arg, PyObject **presult, tstate->exc_info = gen->gi_exc_state.previous_item; gen->gi_exc_state.previous_item = NULL; - assert(tstate->frame == frame->previous); + assert(tstate->cframe->current_frame == frame->previous); /* Don't keep the reference to previous any longer than necessary. It * may keep a chain of frames alive or it could create a reference * cycle. */ @@ -435,9 +435,9 @@ _gen_throw(PyGenObject *gen, int close_on_genexit, will be reported correctly to the user. */ /* XXX We should probably be updating the current frame somewhere in ceval.c. */ - InterpreterFrame *prev = tstate->frame; + InterpreterFrame *prev = tstate->cframe->current_frame; frame->previous = prev; - tstate->frame = frame; + tstate->cframe->current_frame = frame; /* Close the generator that we are currently iterating with 'yield from' or awaiting on with 'await'. */ PyFrameState state = gen->gi_xframe->f_state; @@ -445,7 +445,7 @@ _gen_throw(PyGenObject *gen, int close_on_genexit, ret = _gen_throw((PyGenObject *)yf, close_on_genexit, typ, val, tb); gen->gi_xframe->f_state = state; - tstate->frame = prev; + tstate->cframe->current_frame = prev; frame->previous = NULL; } else { /* `yf` is an iterator or a coroutine-like object. */ diff --git a/Python/ceval.c b/Python/ceval.c index bd01e8159c8e4..4ac0b53dd6e46 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -1561,8 +1561,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr tstate->cframe = &cframe; assert(frame->depth == 0); - /* push frame */ - tstate->frame = frame; + /* Push frame */ + frame->previous = prev_cframe->current_frame; + cframe.current_frame = frame; start_frame: if (_Py_EnterRecursiveCall(tstate, "")) { @@ -1570,7 +1571,8 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto exit_eval_frame; } - assert(frame == tstate->frame); + assert(tstate->cframe == &cframe); + assert(frame == cframe.current_frame); if (cframe.use_tracing) { if (trace_function_entry(tstate, frame)) { @@ -4641,8 +4643,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr goto error; } _PyFrame_SetStackPointer(frame, stack_pointer); + new_frame->previous = frame; + cframe.current_frame = frame = new_frame; new_frame->depth = frame->depth + 1; - tstate->frame = frame = new_frame; goto start_frame; } } @@ -4723,9 +4726,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr STACK_SHRINK(1); Py_DECREF(func); _PyFrame_SetStackPointer(frame, stack_pointer); - new_frame->previous = tstate->frame; + new_frame->previous = frame; + frame = cframe.current_frame = new_frame; new_frame->depth = frame->depth + 1; - tstate->frame = frame = new_frame; goto start_frame; } @@ -5258,11 +5261,12 @@ MISS_WITH_OPARG_COUNTER(BINARY_MULTIPLY) _Py_LeaveRecursiveCall(tstate); if (frame->depth) { - _PyFrame_StackPush(frame->previous, retval); + cframe.current_frame = frame->previous; + _PyFrame_StackPush(cframe.current_frame, retval); if (_PyEvalFrameClearAndPop(tstate, frame)) { retval = NULL; } - frame = tstate->frame; + frame = cframe.current_frame; if (retval == NULL) { assert(_PyErr_Occurred(tstate)); throwflag = 1; @@ -5270,11 +5274,11 @@ MISS_WITH_OPARG_COUNTER(BINARY_MULTIPLY) retval = NULL; goto resume_frame; } - tstate->frame = frame->previous; - /* Restore previous cframe */ + /* Restore previous cframe. */ tstate->cframe = cframe.previous; tstate->cframe->use_tracing = cframe.use_tracing; + assert(tstate->cframe->current_frame == frame->previous); return _Py_CheckFunctionResult(tstate, NULL, retval, __func__); } @@ -5891,8 +5895,6 @@ _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, _PyFrame_Clear(frame, 0); return NULL; } - frame->previous = tstate->frame; - tstate->frame = frame; return frame; } @@ -5906,7 +5908,6 @@ _PyEvalFrameClearAndPop(PyThreadState *tstate, InterpreterFrame * frame) return -1; } --tstate->recursion_depth; - tstate->frame = frame->previous; _PyThreadState_PopFrame(tstate, frame); return 0; } @@ -6518,17 +6519,17 @@ InterpreterFrame * _PyEval_GetFrame(void) { PyThreadState *tstate = _PyThreadState_GET(); - return tstate->frame; + return tstate->cframe->current_frame; } PyFrameObject * PyEval_GetFrame(void) { PyThreadState *tstate = _PyThreadState_GET(); - if (tstate->frame == NULL) { + if (tstate->cframe->current_frame == NULL) { return NULL; } - PyFrameObject *f = _PyFrame_GetFrameObject(tstate->frame); + PyFrameObject *f = _PyFrame_GetFrameObject(tstate->cframe->current_frame); if (f == NULL) { PyErr_Clear(); } @@ -6538,7 +6539,7 @@ PyEval_GetFrame(void) PyObject * _PyEval_GetBuiltins(PyThreadState *tstate) { - InterpreterFrame *frame = tstate->frame; + InterpreterFrame *frame = tstate->cframe->current_frame; if (frame != NULL) { return frame->f_builtins; } @@ -6571,7 +6572,7 @@ PyObject * PyEval_GetLocals(void) { PyThreadState *tstate = _PyThreadState_GET(); - InterpreterFrame *current_frame = tstate->frame; + InterpreterFrame *current_frame = tstate->cframe->current_frame; if (current_frame == NULL) { _PyErr_SetString(tstate, PyExc_SystemError, "frame does not exist"); return NULL; @@ -6590,7 +6591,7 @@ PyObject * PyEval_GetGlobals(void) { PyThreadState *tstate = _PyThreadState_GET(); - InterpreterFrame *current_frame = tstate->frame; + InterpreterFrame *current_frame = tstate->cframe->current_frame; if (current_frame == NULL) { return NULL; } @@ -6601,7 +6602,7 @@ int PyEval_MergeCompilerFlags(PyCompilerFlags *cf) { PyThreadState *tstate = _PyThreadState_GET(); - InterpreterFrame *current_frame = tstate->frame; + InterpreterFrame *current_frame = tstate->cframe->current_frame; int result = cf->cf_flags != 0; if (current_frame != NULL) { @@ -6651,7 +6652,7 @@ PyEval_GetFuncDesc(PyObject *func) #define C_TRACE(x, call) \ if (use_tracing && tstate->c_profilefunc) { \ if (call_trace(tstate->c_profilefunc, tstate->c_profileobj, \ - tstate, tstate->frame, \ + tstate, tstate->cframe->current_frame, \ PyTrace_C_CALL, func)) { \ x = NULL; \ } \ @@ -6661,13 +6662,13 @@ if (use_tracing && tstate->c_profilefunc) { \ if (x == NULL) { \ call_trace_protected(tstate->c_profilefunc, \ tstate->c_profileobj, \ - tstate, tstate->frame, \ + tstate, tstate->cframe->current_frame, \ PyTrace_C_EXCEPTION, func); \ /* XXX should pass (type, value, tb) */ \ } else { \ if (call_trace(tstate->c_profilefunc, \ tstate->c_profileobj, \ - tstate, tstate->frame, \ + tstate, tstate->cframe->current_frame, \ PyTrace_C_RETURN, func)) { \ Py_DECREF(x); \ x = NULL; \ diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 9ce845ca61d21..3ccf32ab1bb4d 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -2031,7 +2031,7 @@ Py_EndInterpreter(PyThreadState *tstate) if (tstate != _PyThreadState_GET()) { Py_FatalError("thread is not current"); } - if (tstate->frame != NULL) { + if (tstate->cframe->current_frame != NULL) { Py_FatalError("thread still has a frame"); } interp->finalizing = 1; diff --git a/Python/pystate.c b/Python/pystate.c index 7804e17a064e1..114f91559d9cc 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -636,12 +636,12 @@ new_threadstate(PyInterpreterState *interp, int init) tstate->interp = interp; - tstate->frame = NULL; tstate->recursion_depth = 0; tstate->recursion_headroom = 0; tstate->stackcheck_counter = 0; tstate->tracing = 0; tstate->root_cframe.use_tracing = 0; + tstate->root_cframe.current_frame = NULL; tstate->cframe = &tstate->root_cframe; tstate->gilstate_counter = 0; tstate->async_exc = NULL; @@ -861,7 +861,7 @@ PyThreadState_Clear(PyThreadState *tstate) { int verbose = _PyInterpreterState_GetConfig(tstate->interp)->verbose; - if (verbose && tstate->frame != NULL) { + if (verbose && tstate->cframe->current_frame != NULL) { /* bpo-20526: After the main thread calls _PyRuntimeState_SetFinalizing() in Py_FinalizeEx(), threads must exit when trying to take the GIL. If a thread exit in the middle of @@ -1134,10 +1134,10 @@ PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate) { assert(tstate != NULL); - if (tstate->frame == NULL) { + if (tstate->cframe->current_frame == NULL) { return NULL; } - PyFrameObject *frame = _PyFrame_GetFrameObject(tstate->frame); + PyFrameObject *frame = _PyFrame_GetFrameObject(tstate->cframe->current_frame); if (frame == NULL) { PyErr_Clear(); } @@ -1277,7 +1277,7 @@ _PyThread_CurrentFrames(void) for (i = runtime->interpreters.head; i != NULL; i = i->next) { PyThreadState *t; for (t = i->tstate_head; t != NULL; t = t->next) { - InterpreterFrame *frame = t->frame; + InterpreterFrame *frame = t->cframe->current_frame; if (frame == NULL) { continue; } diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 5e663c17c79b8..27937a03e8968 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1813,7 +1813,7 @@ sys__getframe_impl(PyObject *module, int depth) /*[clinic end generated code: output=d438776c04d59804 input=c1be8a6464b11ee5]*/ { PyThreadState *tstate = _PyThreadState_GET(); - InterpreterFrame *frame = tstate->frame; + InterpreterFrame *frame = tstate->cframe->current_frame; if (_PySys_Audit(tstate, "sys._getframe", NULL) < 0) { return NULL; diff --git a/Python/traceback.c b/Python/traceback.c index b18cbb91ce8ed..22a0922c255b3 100644 --- a/Python/traceback.c +++ b/Python/traceback.c @@ -1082,7 +1082,7 @@ dump_traceback(int fd, PyThreadState *tstate, int write_header) PUTS(fd, "Stack (most recent call first):\n"); } - frame = tstate->frame; + frame = tstate->cframe->current_frame; if (frame == NULL) { PUTS(fd, " \n"); return; From webhook-mailer at python.org Thu Oct 28 11:02:57 2021 From: webhook-mailer at python.org (markshannon) Date: Thu, 28 Oct 2021 15:02:57 -0000 Subject: [Python-checkins] bpo-44525: Add recursive checks for `CALL_FUNCTION_BUILTIN_O` (GH-29271) Message-ID: https://github.com/python/cpython/commit/0a1a36b74bdf8da286924a1c9652853b1c46f536 commit: 0a1a36b74bdf8da286924a1c9652853b1c46f536 branch: main author: Ken Jin <28750310+Fidget-Spinner at users.noreply.github.com> committer: markshannon date: 2021-10-28T16:02:34+01:00 summary: bpo-44525: Add recursive checks for `CALL_FUNCTION_BUILTIN_O` (GH-29271) files: M Python/ceval.c diff --git a/Python/ceval.c b/Python/ceval.c index 4ac0b53dd6e46..d52ca9c65db22 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -4742,8 +4742,14 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr STAT_INC(CALL_FUNCTION, hit); PyCFunction cfunc = PyCFunction_GET_FUNCTION(callable); + // This is slower but CPython promises to check all non-vectorcall + // function calls. + if (_Py_EnterRecursiveCall(tstate, " while calling a Python object")) { + goto error; + } PyObject *arg = POP(); PyObject *res = cfunc(PyCFunction_GET_SELF(callable), arg); + _Py_LeaveRecursiveCall(tstate); assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); /* Clear the stack of the function object. */ From webhook-mailer at python.org Thu Oct 28 11:15:08 2021 From: webhook-mailer at python.org (markshannon) Date: Thu, 28 Oct 2021 15:15:08 -0000 Subject: [Python-checkins] bpo-45256: Rationalize code around Python-to-Python calls a bit. (GH-29235) Message-ID: https://github.com/python/cpython/commit/7f61d9d84843e3445f62eb00c47902f0daa30a72 commit: 7f61d9d84843e3445f62eb00c47902f0daa30a72 branch: main author: Mark Shannon committer: markshannon date: 2021-10-28T16:14:59+01:00 summary: bpo-45256: Rationalize code around Python-to-Python calls a bit. (GH-29235) files: M Include/internal/pycore_frame.h M Objects/frameobject.c M Python/ceval.c M Python/pystate.c diff --git a/Include/internal/pycore_frame.h b/Include/internal/pycore_frame.h index 7e63f584eb3b0..b025ca0b8d766 100644 --- a/Include/internal/pycore_frame.h +++ b/Include/internal/pycore_frame.h @@ -152,6 +152,21 @@ _PyFrame_LocalsToFast(InterpreterFrame *frame, int clear); InterpreterFrame *_PyThreadState_PushFrame( PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals); +extern InterpreterFrame * +_PyThreadState_BumpFramePointerSlow(PyThreadState *tstate, size_t size); + +static inline InterpreterFrame * +_PyThreadState_BumpFramePointer(PyThreadState *tstate, size_t size) +{ + PyObject **base = tstate->datastack_top; + PyObject **top = base + size; + if (top < tstate->datastack_limit) { + tstate->datastack_top = top; + return (InterpreterFrame *)base; + } + return _PyThreadState_BumpFramePointerSlow(tstate, size); +} + void _PyThreadState_PopFrame(PyThreadState *tstate, InterpreterFrame *frame); #ifdef __cplusplus diff --git a/Objects/frameobject.c b/Objects/frameobject.c index ffe19b3d99400..09857c7fa007d 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -786,16 +786,15 @@ allocate_heap_frame(PyFrameConstructor *con, PyObject *locals) { PyCodeObject *code = (PyCodeObject *)con->fc_code; int size = code->co_nlocalsplus+code->co_stacksize + FRAME_SPECIALS_SIZE; - PyObject **localsarray = PyMem_Malloc(sizeof(PyObject *)*size); - if (localsarray == NULL) { + InterpreterFrame *frame = (InterpreterFrame *)PyMem_Malloc(sizeof(PyObject *)*size); + if (frame == NULL) { PyErr_NoMemory(); return NULL; } - for (Py_ssize_t i=0; i < code->co_nlocalsplus; i++) { - localsarray[i] = NULL; - } - InterpreterFrame *frame = (InterpreterFrame *)(localsarray + code->co_nlocalsplus); _PyFrame_InitializeSpecials(frame, con, locals, code->co_nlocalsplus); + for (Py_ssize_t i = 0; i < code->co_nlocalsplus; i++) { + frame->localsplus[i] = NULL; + } return frame; } diff --git a/Python/ceval.c b/Python/ceval.c index d52ca9c65db22..c9f6638afa0e4 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -101,7 +101,7 @@ static int get_exception_handler(PyCodeObject *, int, int*, int*, int*); static InterpreterFrame * _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals, PyObject* const* args, - size_t argcount, PyObject *kwnames, int steal_args); + size_t argcount, PyObject *kwnames); static int _PyEvalFrameClearAndPop(PyThreadState *tstate, InterpreterFrame * frame); @@ -4633,7 +4633,7 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr InterpreterFrame *new_frame = _PyEvalFramePushAndInit( tstate, PyFunction_AS_FRAME_CONSTRUCTOR(function), locals, stack_pointer, - nargs, kwnames, 1); + nargs, kwnames); STACK_SHRINK(postcall_shrink); // The frame has stolen all the arguments from the stack, // so there is no need to clean them up. @@ -4708,11 +4708,14 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr /* PEP 523 */ DEOPT_IF(tstate->interp->eval_frame != NULL, CALL_FUNCTION); STAT_INC(CALL_FUNCTION, hit); - InterpreterFrame *new_frame = _PyThreadState_PushFrame( - tstate, PyFunction_AS_FRAME_CONSTRUCTOR(func), NULL); + PyCodeObject *code = (PyCodeObject *)func->func_code; + size_t size = code->co_nlocalsplus + code->co_stacksize + FRAME_SPECIALS_SIZE; + InterpreterFrame *new_frame = _PyThreadState_BumpFramePointer(tstate, size); if (new_frame == NULL) { goto error; } + _PyFrame_InitializeSpecials(new_frame, PyFunction_AS_FRAME_CONSTRUCTOR(func), + NULL, code->co_nlocalsplus); STACK_SHRINK(argcount); for (int i = 0; i < argcount; i++) { new_frame->localsplus[i] = stack_pointer[i]; @@ -4723,6 +4726,9 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, InterpreterFrame *frame, int thr Py_INCREF(def); new_frame->localsplus[argcount+i] = def; } + for (int i = argcount+deflen; i < code->co_nlocalsplus; i++) { + new_frame->localsplus[i] = NULL; + } STACK_SHRINK(1); Py_DECREF(func); _PyFrame_SetStackPointer(frame, stack_pointer); @@ -5595,7 +5601,7 @@ get_exception_handler(PyCodeObject *code, int index, int *level, int *handler, i static int initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, PyObject **localsplus, PyObject *const *args, - Py_ssize_t argcount, PyObject *kwnames, int steal_args) + Py_ssize_t argcount, PyObject *kwnames) { PyCodeObject *co = (PyCodeObject*)con->fc_code; const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount; @@ -5629,9 +5635,6 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, } for (j = 0; j < n; j++) { PyObject *x = args[j]; - if (!steal_args) { - Py_INCREF(x); - } assert(localsplus[j] == NULL); localsplus[j] = x; } @@ -5639,11 +5642,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, /* Pack other positional arguments into the *args argument */ if (co->co_flags & CO_VARARGS) { PyObject *u = NULL; - if (steal_args) { - u = _PyTuple_FromArraySteal(args + n, argcount - n); - } else { - u = _PyTuple_FromArray(args + n, argcount - n); - } + u = _PyTuple_FromArraySteal(args + n, argcount - n); if (u == NULL) { goto fail_post_positional; } @@ -5652,10 +5651,8 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, } else if (argcount > n) { /* Too many postional args. Error is reported later */ - if (steal_args) { - for (j = n; j < argcount; j++) { - Py_DECREF(args[j]); - } + for (j = n; j < argcount; j++) { + Py_DECREF(args[j]); } } @@ -5717,19 +5714,15 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (PyDict_SetItem(kwdict, keyword, value) == -1) { goto kw_fail; } - if (steal_args) { - Py_DECREF(value); - } + Py_DECREF(value); continue; kw_fail: - if (steal_args) { - for (;i < kwcount; i++) { - PyObject *value = args[i+argcount]; - Py_DECREF(value); - } + for (;i < kwcount; i++) { + PyObject *value = args[i+argcount]; + Py_DECREF(value); } - goto fail_noclean; + goto fail_post_args; kw_found: if (localsplus[j] != NULL) { @@ -5738,9 +5731,6 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, con->fc_qualname, keyword); goto kw_fail; } - if (!steal_args) { - Py_INCREF(value); - } localsplus[j] = value; } } @@ -5749,7 +5739,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if ((argcount > co->co_argcount) && !(co->co_flags & CO_VARARGS)) { too_many_positional(tstate, co, argcount, con->fc_defaults, localsplus, con->fc_qualname); - goto fail_noclean; + goto fail_post_args; } /* Add missing positional arguments (copy default values from defs) */ @@ -5765,7 +5755,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (missing) { missing_arguments(tstate, co, missing, defcount, localsplus, con->fc_qualname); - goto fail_noclean; + goto fail_post_args; } if (n > m) i = n - m; @@ -5798,7 +5788,7 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, continue; } else if (_PyErr_Occurred(tstate)) { - goto fail_noclean; + goto fail_post_args; } } missing++; @@ -5806,35 +5796,31 @@ initialize_locals(PyThreadState *tstate, PyFrameConstructor *con, if (missing) { missing_arguments(tstate, co, missing, -1, localsplus, con->fc_qualname); - goto fail_noclean; + goto fail_post_args; } } - /* Copy closure variables to free variables */ for (i = 0; i < co->co_nfreevars; ++i) { PyObject *o = PyTuple_GET_ITEM(con->fc_closure, i); Py_INCREF(o); localsplus[co->co_nlocals + co->co_nplaincellvars + i] = o; } - return 0; fail_pre_positional: - if (steal_args) { - for (j = 0; j < argcount; j++) { - Py_DECREF(args[j]); - } + for (j = 0; j < argcount; j++) { + Py_DECREF(args[j]); } /* fall through */ fail_post_positional: - if (steal_args) { - Py_ssize_t kwcount = kwnames != NULL ? PyTuple_GET_SIZE(kwnames) : 0; + if (kwnames) { + Py_ssize_t kwcount = PyTuple_GET_SIZE(kwnames); for (j = argcount; j < argcount+kwcount; j++) { Py_DECREF(args[j]); } } /* fall through */ -fail_noclean: +fail_post_args: return -1; } @@ -5850,21 +5836,34 @@ make_coro_frame(PyThreadState *tstate, int size = code->co_nlocalsplus+code->co_stacksize + FRAME_SPECIALS_SIZE; InterpreterFrame *frame = (InterpreterFrame *)PyMem_Malloc(sizeof(PyObject *)*size); if (frame == NULL) { - PyErr_NoMemory(); - return NULL; + goto fail_no_memory; } - for (Py_ssize_t i=0; i < code->co_nlocalsplus; i++) { + _PyFrame_InitializeSpecials(frame, con, locals, code->co_nlocalsplus); + for (int i = 0; i < code->co_nlocalsplus; i++) { frame->localsplus[i] = NULL; } - _PyFrame_InitializeSpecials(frame, con, locals, code->co_nlocalsplus); assert(frame->frame_obj == NULL); - if (initialize_locals(tstate, con, frame->localsplus, args, argcount, kwnames, 0)) { + if (initialize_locals(tstate, con, frame->localsplus, args, argcount, kwnames)) { _PyFrame_Clear(frame, 1); return NULL; } return frame; +fail_no_memory: + /* Consume the references */ + for (Py_ssize_t i = 0; i < argcount; i++) { + Py_DECREF(args[i]); + } + if (kwnames) { + Py_ssize_t kwcount = PyTuple_GET_SIZE(kwnames); + for (Py_ssize_t i = 0; i < kwcount; i++) { + Py_DECREF(args[i+argcount]); + } + } + PyErr_NoMemory(); + return NULL; } +/* Consumes all the references to the args */ static PyObject * make_coro(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals, @@ -5880,28 +5879,44 @@ make_coro(PyThreadState *tstate, PyFrameConstructor *con, if (gen == NULL) { return NULL; } - return gen; } -// If *steal_args* is set, the function will steal the references to all the arguments. -// In case of error, the function returns null and if *steal_args* is set, the caller -// will still own all the arguments. +/* Consumes all the references to the args */ static InterpreterFrame * _PyEvalFramePushAndInit(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals, PyObject* const* args, - size_t argcount, PyObject *kwnames, int steal_args) + size_t argcount, PyObject *kwnames) { - InterpreterFrame * frame = _PyThreadState_PushFrame(tstate, con, locals); + PyCodeObject * code = (PyCodeObject *)con->fc_code; + size_t size = code->co_nlocalsplus + code->co_stacksize + FRAME_SPECIALS_SIZE; + InterpreterFrame *frame = _PyThreadState_BumpFramePointer(tstate, size); if (frame == NULL) { - return NULL; + goto fail; } - PyObject **localsarray = _PyFrame_GetLocalsArray(frame); - if (initialize_locals(tstate, con, localsarray, args, argcount, kwnames, steal_args)) { + _PyFrame_InitializeSpecials(frame, con, locals, code->co_nlocalsplus); + PyObject **localsarray = &frame->localsplus[0]; + for (int i = 0; i < code->co_nlocalsplus; i++) { + localsarray[i] = NULL; + } + if (initialize_locals(tstate, con, localsarray, args, argcount, kwnames)) { _PyFrame_Clear(frame, 0); return NULL; } return frame; +fail: + /* Consume the references */ + for (size_t i = 0; i < argcount; i++) { + Py_DECREF(args[i]); + } + if (kwnames) { + Py_ssize_t kwcount = PyTuple_GET_SIZE(kwnames); + for (Py_ssize_t i = 0; i < kwcount; i++) { + Py_DECREF(args[i+argcount]); + } + } + PyErr_NoMemory(); + return NULL; } static int @@ -5925,13 +5940,25 @@ _PyEval_Vector(PyThreadState *tstate, PyFrameConstructor *con, PyObject *kwnames) { PyCodeObject *code = (PyCodeObject *)con->fc_code; + /* _PyEvalFramePushAndInit and make_coro consume + * all the references to their arguments + */ + for (size_t i = 0; i < argcount; i++) { + Py_INCREF(args[i]); + } + if (kwnames) { + Py_ssize_t kwcount = PyTuple_GET_SIZE(kwnames); + for (Py_ssize_t i = 0; i < kwcount; i++) { + Py_INCREF(args[i+argcount]); + } + } int is_coro = code->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR); if (is_coro) { return make_coro(tstate, con, locals, args, argcount, kwnames); } InterpreterFrame *frame = _PyEvalFramePushAndInit( - tstate, con, locals, args, argcount, kwnames, 0); + tstate, con, locals, args, argcount, kwnames); if (frame == NULL) { return NULL; } diff --git a/Python/pystate.c b/Python/pystate.c index 114f91559d9cc..a9ed08a7e34be 100644 --- a/Python/pystate.c +++ b/Python/pystate.c @@ -2065,12 +2065,8 @@ push_chunk(PyThreadState *tstate, int size) } InterpreterFrame * -_PyThreadState_PushFrame(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals) +_PyThreadState_BumpFramePointerSlow(PyThreadState *tstate, size_t size) { - PyCodeObject *code = (PyCodeObject *)con->fc_code; - int nlocalsplus = code->co_nlocalsplus; - size_t size = nlocalsplus + code->co_stacksize + - FRAME_SPECIALS_SIZE; assert(size < INT_MAX/sizeof(PyObject *)); PyObject **base = tstate->datastack_top; PyObject **top = base + size; @@ -2083,7 +2079,21 @@ _PyThreadState_PushFrame(PyThreadState *tstate, PyFrameConstructor *con, PyObjec else { tstate->datastack_top = top; } - InterpreterFrame *frame = (InterpreterFrame *)base; + return (InterpreterFrame *)base; +} + + +InterpreterFrame * +_PyThreadState_PushFrame(PyThreadState *tstate, PyFrameConstructor *con, PyObject *locals) +{ + PyCodeObject *code = (PyCodeObject *)con->fc_code; + int nlocalsplus = code->co_nlocalsplus; + size_t size = nlocalsplus + code->co_stacksize + + FRAME_SPECIALS_SIZE; + InterpreterFrame *frame = _PyThreadState_BumpFramePointer(tstate, size); + if (frame == NULL) { + return NULL; + } _PyFrame_InitializeSpecials(frame, con, locals, nlocalsplus); for (int i=0; i < nlocalsplus; i++) { frame->localsplus[i] = NULL; From webhook-mailer at python.org Thu Oct 28 12:02:12 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 16:02:12 -0000 Subject: [Python-checkins] [3.9] bpo-39679: Fix `singledispatchmethod` `classmethod`/`staticmethod` bug (GH-29087) Message-ID: https://github.com/python/cpython/commit/97388c204b557f30e48a2b2ef826868702204cf2 commit: 97388c204b557f30e48a2b2ef826868702204cf2 branch: 3.9 author: Alex Waygood committer: ambv date: 2021-10-28T18:02:04+02:00 summary: [3.9] bpo-39679: Fix `singledispatchmethod` `classmethod`/`staticmethod` bug (GH-29087) This commit fixes a bug in the 3.9 branch where stacking `@functools.singledispatchmethod` on top of `@classmethod` or `@staticmethod` caused an exception to be raised if the method was registered using type-annotations rather than `@method.register(int)`. Tests for this scenario were added to the 3.11 and 3.10 branches in #29034 and #29072; this commit also backports those tests to the 3.9 branch. Co-authored-by: Yurii Karabas <1998uriyyo at gmail.com> Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Library/2021-10-20-10-07-44.bpo-39679.nVYJJ3.rst M Lib/functools.py M Lib/test/test_functools.py diff --git a/Lib/functools.py b/Lib/functools.py index 97744a869563d..5054e281ad281 100644 --- a/Lib/functools.py +++ b/Lib/functools.py @@ -906,6 +906,12 @@ def register(self, cls, method=None): Registers a new implementation for the given *cls* on a *generic_method*. """ + # bpo-39679: in Python <= 3.9, classmethods and staticmethods don't + # inherit __annotations__ of the wrapped function (fixed in 3.10+ as + # a side-effect of bpo-43682) but we need that for annotation-derived + # singledispatches. So we add that just-in-time here. + if isinstance(cls, (staticmethod, classmethod)): + cls.__annotations__ = getattr(cls.__func__, '__annotations__', {}) return self.dispatcher.register(cls, func=method) def __get__(self, obj, cls=None): diff --git a/Lib/test/test_functools.py b/Lib/test/test_functools.py index 987020ea007fa..96e93ed8eab34 100644 --- a/Lib/test/test_functools.py +++ b/Lib/test/test_functools.py @@ -2427,6 +2427,48 @@ def _(self, arg: str): self.assertEqual(a.t(''), "str") self.assertEqual(a.t(0.0), "base") + def test_staticmethod_type_ann_register(self): + class A: + @functools.singledispatchmethod + @staticmethod + def t(arg): + return arg + @t.register + @staticmethod + def _(arg: int): + return isinstance(arg, int) + @t.register + @staticmethod + def _(arg: str): + return isinstance(arg, str) + a = A() + + self.assertTrue(A.t(0)) + self.assertTrue(A.t('')) + self.assertEqual(A.t(0.0), 0.0) + + def test_classmethod_type_ann_register(self): + class A: + def __init__(self, arg): + self.arg = arg + + @functools.singledispatchmethod + @classmethod + def t(cls, arg): + return cls("base") + @t.register + @classmethod + def _(cls, arg: int): + return cls("int") + @t.register + @classmethod + def _(cls, arg: str): + return cls("str") + + self.assertEqual(A.t(0).arg, "int") + self.assertEqual(A.t('').arg, "str") + self.assertEqual(A.t(0.0).arg, "base") + def test_invalid_registrations(self): msg_prefix = "Invalid first argument to `register()`: " msg_suffix = ( diff --git a/Misc/NEWS.d/next/Library/2021-10-20-10-07-44.bpo-39679.nVYJJ3.rst b/Misc/NEWS.d/next/Library/2021-10-20-10-07-44.bpo-39679.nVYJJ3.rst new file mode 100644 index 0000000000000..b0656aac51677 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-20-10-07-44.bpo-39679.nVYJJ3.rst @@ -0,0 +1,3 @@ +Fix bug in :class:`functools.singledispatchmethod` that caused it to fail +when attempting to register a :func:`classmethod` or :func:`staticmethod` +using type annotations. Patch contributed by Alex Waygood. From webhook-mailer at python.org Thu Oct 28 12:14:45 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Thu, 28 Oct 2021 16:14:45 -0000 Subject: [Python-checkins] bpo-45629: Add a test for the "freeze" tool. (gh-29222) Message-ID: https://github.com/python/cpython/commit/13d9205f4057eeeef80a25d410ad123876dc60cd commit: 13d9205f4057eeeef80a25d410ad123876dc60cd branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-28T10:14:37-06:00 summary: bpo-45629: Add a test for the "freeze" tool. (gh-29222) The "freeze" tool has been part of the repo for a long time. However, it hasn't had any tests in the test suite to guard against regressions. We add such a test here. This is especially important as there has been a lot of change recently related to frozen modules, with more to come. Note that as part of the test we build Python out-of-tree and install it in a temp dir. https://bugs.python.org/issue45629 files: A Lib/test/test_tools/test_freeze.py A Tools/freeze/test/freeze.py M .gitignore M Lib/test/support/__init__.py diff --git a/.gitignore b/.gitignore index c3fc748ea97c3..b2ad76689f12b 100644 --- a/.gitignore +++ b/.gitignore @@ -120,6 +120,7 @@ Tools/unicode/data/ Tools/msi/obj Tools/ssl/amd64 Tools/ssl/win32 +Tools/freeze/test/outdir # The frozen modules are always generated by the build so we don't # keep them in the repo. Also see Tools/scripts/freeze_modules.py. diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 85fd74126b5f4..fc3c99ec0ecc3 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -372,6 +372,17 @@ def wrapper(*args, **kw): return decorator +def skip_if_buildbot(reason=None): + """Decorator raising SkipTest if running on a buildbot.""" + if not reason: + reason = 'not suitable for buildbots' + if sys.platform == 'win32': + isbuildbot = os.environ.get('USERNAME') == 'Buildbot' + else: + isbuildbot = os.environ.get('USER') == 'buildbot' + return unittest.skipIf(isbuildbot, reason) + + def system_must_validate_cert(f): """Skip the test on TLS certificate validation failures.""" @functools.wraps(f) diff --git a/Lib/test/test_tools/test_freeze.py b/Lib/test/test_tools/test_freeze.py new file mode 100644 index 0000000000000..392a776f042e4 --- /dev/null +++ b/Lib/test/test_tools/test_freeze.py @@ -0,0 +1,29 @@ +"""Sanity-check tests for the "freeze" tool.""" + +import sys +import textwrap +import unittest + +from test import support + +from . import imports_under_tool, skip_if_missing +skip_if_missing('freeze') +with imports_under_tool('freeze', 'test'): + import freeze as helper + + + at unittest.skipIf(sys.platform.startswith('win'), 'not supported on Windows') + at support.skip_if_buildbot('not all buildbots have enough space') +class TestFreeze(unittest.TestCase): + + def test_freeze_simple_script(self): + script = textwrap.dedent(""" + import sys + print('running...') + sys.exit(0) + """) + outdir, scriptfile, python = helper.prepare(script) + + executable = helper.freeze(python, scriptfile, outdir) + text = helper.run(executable) + self.assertEqual(text, 'running...') diff --git a/Tools/freeze/test/freeze.py b/Tools/freeze/test/freeze.py new file mode 100644 index 0000000000000..18a5d27cebf2e --- /dev/null +++ b/Tools/freeze/test/freeze.py @@ -0,0 +1,194 @@ +import os +import os.path +import re +import shlex +import shutil +import subprocess + + +TESTS_DIR = os.path.dirname(__file__) +TOOL_ROOT = os.path.dirname(TESTS_DIR) +SRCDIR = os.path.dirname(os.path.dirname(TOOL_ROOT)) + +MAKE = shutil.which('make') +GIT = shutil.which('git') +FREEZE = os.path.join(TOOL_ROOT, 'freeze.py') +OUTDIR = os.path.join(TESTS_DIR, 'outdir') + + +class UnsupportedError(Exception): + """The operation isn't supported.""" + + +def _run_quiet(cmd, cwd=None): + #print(f'# {" ".join(shlex.quote(a) for a in cmd)}') + return subprocess.run( + cmd, + cwd=cwd, + capture_output=True, + text=True, + check=True, + ) + + +def _run_stdout(cmd, cwd=None): + proc = _run_quiet(cmd, cwd) + return proc.stdout.strip() + + +def find_opt(args, name): + opt = f'--{name}' + optstart = f'{opt}=' + for i, arg in enumerate(args): + if arg == opt or arg.startswith(optstart): + return i + return -1 + + +def ensure_opt(args, name, value): + opt = f'--{name}' + pos = find_opt(args, name) + if value is None: + if pos < 0: + args.append(opt) + else: + args[pos] = opt + elif pos < 0: + args.extend([opt, value]) + else: + arg = args[pos] + if arg == opt: + if pos == len(args) - 1: + raise NotImplementedError((args, opt)) + args[pos + 1] = value + else: + args[pos] = f'{opt}={value}' + + +def git_copy_repo(newroot, oldroot): + if not GIT: + raise UnsupportedError('git') + + if os.path.exists(newroot): + print(f'updating copied repo {newroot}...') + if newroot == SRCDIR: + raise Exception('this probably isn\'t what you wanted') + _run_quiet([GIT, 'clean', '-d', '-f'], newroot) + _run_quiet([GIT, 'reset'], newroot) + _run_quiet([GIT, 'checkout', '.'], newroot) + _run_quiet([GIT, 'pull', '-f', oldroot], newroot) + else: + print(f'copying repo into {newroot}...') + _run_quiet([GIT, 'clone', oldroot, newroot]) + + # Copy over any uncommited files. + text = _run_stdout([GIT, 'status', '-s'], oldroot) + for line in text.splitlines(): + _, _, relfile = line.strip().partition(' ') + relfile = relfile.strip() + isdir = relfile.endswith(os.path.sep) + relfile = relfile.rstrip(os.path.sep) + srcfile = os.path.join(oldroot, relfile) + dstfile = os.path.join(newroot, relfile) + os.makedirs(os.path.dirname(dstfile), exist_ok=True) + if isdir: + shutil.copytree(srcfile, dstfile, dirs_exist_ok=True) + else: + shutil.copy2(srcfile, dstfile) + + +def get_makefile_var(builddir, name): + regex = re.compile(rf'^{name} *=\s*(.*?)\s*$') + filename = os.path.join(builddir, 'Makefile') + try: + infile = open(filename) + except FileNotFoundError: + return None + with infile: + for line in infile: + m = regex.match(line) + if m: + value, = m.groups() + return value or '' + return None + + +def get_config_var(builddir, name): + python = os.path.join(builddir, 'python') + if os.path.isfile(python): + cmd = [python, '-c', + f'import sysconfig; print(sysconfig.get_config_var("{name}"))'] + try: + return _run_stdout(cmd) + except subprocess.CalledProcessError: + pass + return get_makefile_var(builddir, name) + + +################################## +# freezing + +def prepare(script=None, outdir=None): + if not outdir: + outdir = OUTDIR + os.makedirs(outdir, exist_ok=True) + + # Write the script to disk. + if script: + scriptfile = os.path.join(outdir, 'app.py') + with open(scriptfile, 'w') as outfile: + outfile.write(script) + + # Make a copy of the repo to avoid affecting the current build. + srcdir = os.path.join(outdir, 'cpython') + git_copy_repo(srcdir, SRCDIR) + + # We use an out-of-tree build (instead of srcdir). + builddir = os.path.join(outdir, 'python-build') + os.makedirs(builddir, exist_ok=True) + + # Run configure. + print(f'configuring python in {builddir}...') + cmd = [ + os.path.join(srcdir, 'configure'), + *shlex.split(get_config_var(builddir, 'CONFIG_ARGS') or ''), + ] + ensure_opt(cmd, 'cache-file', os.path.join(outdir, 'python-config.cache')) + prefix = os.path.join(outdir, 'python-installation') + ensure_opt(cmd, 'prefix', prefix) + _run_quiet(cmd, builddir) + + if not MAKE: + raise UnsupportedError('make') + + # Build python. + print('building python...') + if os.path.exists(os.path.join(srcdir, 'Makefile')): + # Out-of-tree builds require a clean srcdir. + _run_quiet([MAKE, '-C', srcdir, 'clean']) + _run_quiet([MAKE, '-C', builddir, '-j8']) + + # Install the build. + print(f'installing python into {prefix}...') + _run_quiet([MAKE, '-C', builddir, '-j8', 'install']) + python = os.path.join(prefix, 'bin', 'python3') + + return outdir, scriptfile, python + + +def freeze(python, scriptfile, outdir): + if not MAKE: + raise UnsupportedError('make') + + print(f'freezing {scriptfile}...') + os.makedirs(outdir, exist_ok=True) + _run_quiet([python, FREEZE, '-o', outdir, scriptfile], outdir) + _run_quiet([MAKE, '-C', os.path.dirname(scriptfile)]) + + name = os.path.basename(scriptfile).rpartition('.')[0] + executable = os.path.join(outdir, name) + return executable + + +def run(executable): + return _run_stdout([executable]) From webhook-mailer at python.org Thu Oct 28 12:35:52 2021 From: webhook-mailer at python.org (markshannon) Date: Thu, 28 Oct 2021 16:35:52 -0000 Subject: [Python-checkins] Store actual ints, not pointers to them in the interpreter state. (GH-29274) Message-ID: https://github.com/python/cpython/commit/4fc68560ea0d506c152a82c48c162bfe002f34a8 commit: 4fc68560ea0d506c152a82c48c162bfe002f34a8 branch: main author: Mark Shannon committer: markshannon date: 2021-10-28T17:35:43+01:00 summary: Store actual ints, not pointers to them in the interpreter state. (GH-29274) files: M Include/internal/pycore_interp.h M Include/internal/pycore_long.h M Include/internal/pycore_pylifecycle.h M Objects/longobject.c M Python/pylifecycle.c diff --git a/Include/internal/pycore_interp.h b/Include/internal/pycore_interp.h index c16f0a4b5e643..8bd3dc064eea3 100644 --- a/Include/internal/pycore_interp.h +++ b/Include/internal/pycore_interp.h @@ -335,7 +335,7 @@ struct _is { The integers that are preallocated are those in the range -_PY_NSMALLNEGINTS (inclusive) to _PY_NSMALLPOSINTS (not inclusive). */ - PyLongObject* small_ints[_PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS]; + PyLongObject small_ints[_PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS]; struct _Py_bytes_state bytes; struct _Py_unicode_state unicode; struct _Py_float_state float_state; diff --git a/Include/internal/pycore_long.h b/Include/internal/pycore_long.h index 8bdf8e5736d20..773025b4a5add 100644 --- a/Include/internal/pycore_long.h +++ b/Include/internal/pycore_long.h @@ -17,7 +17,7 @@ static inline PyObject* __PyLong_GetSmallInt_internal(int value) PyInterpreterState *interp = _PyInterpreterState_GET(); assert(-_PY_NSMALLNEGINTS <= value && value < _PY_NSMALLPOSINTS); size_t index = _PY_NSMALLNEGINTS + value; - PyObject *obj = (PyObject*)interp->small_ints[index]; + PyObject *obj = (PyObject*)&interp->small_ints[index]; // _PyLong_GetZero(), _PyLong_GetOne() and get_small_int() must not be // called before _PyLong_Init() nor after _PyLong_Fini(). assert(obj != NULL); diff --git a/Include/internal/pycore_pylifecycle.h b/Include/internal/pycore_pylifecycle.h index 53b94748b32e9..5e0f36ab2ae49 100644 --- a/Include/internal/pycore_pylifecycle.h +++ b/Include/internal/pycore_pylifecycle.h @@ -53,7 +53,7 @@ extern PyStatus _PyUnicode_Init(PyInterpreterState *interp); extern PyStatus _PyUnicode_InitTypes(void); extern PyStatus _PyBytes_Init(PyInterpreterState *interp); extern int _PyStructSequence_Init(void); -extern int _PyLong_Init(PyInterpreterState *interp); +extern void _PyLong_Init(PyInterpreterState *interp); extern int _PyLong_InitTypes(void); extern PyStatus _PyTuple_Init(PyInterpreterState *interp); extern PyStatus _PyFaulthandler_Init(int enable); diff --git a/Objects/longobject.c b/Objects/longobject.c index 5325d1852bc02..b7392e50e7f6c 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -5827,24 +5827,17 @@ PyLong_GetInfo(void) return int_info; } -int +void _PyLong_Init(PyInterpreterState *interp) { for (Py_ssize_t i=0; i < NSMALLNEGINTS + NSMALLPOSINTS; i++) { sdigit ival = (sdigit)i - NSMALLNEGINTS; int size = (ival < 0) ? -1 : ((ival == 0) ? 0 : 1); - - PyLongObject *v = _PyLong_New(1); - if (!v) { - return -1; - } - - Py_SET_SIZE(v, size); - v->ob_digit[0] = (digit)abs(ival); - - interp->small_ints[i] = v; + interp->small_ints[i].ob_base.ob_base.ob_refcnt = 1; + interp->small_ints[i].ob_base.ob_base.ob_type = &PyLong_Type; + interp->small_ints[i].ob_base.ob_size = size; + interp->small_ints[i].ob_digit[0] = (digit)abs(ival); } - return 0; } @@ -5863,7 +5856,5 @@ _PyLong_InitTypes(void) void _PyLong_Fini(PyInterpreterState *interp) { - for (Py_ssize_t i = 0; i < NSMALLNEGINTS + NSMALLPOSINTS; i++) { - Py_CLEAR(interp->small_ints[i]); - } + (void)interp; } diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 3ccf32ab1bb4d..7e6060e4ebcf7 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -659,9 +659,7 @@ pycore_init_singletons(PyInterpreterState *interp) { PyStatus status; - if (_PyLong_Init(interp) < 0) { - return _PyStatus_ERR("can't init longs"); - } + _PyLong_Init(interp); if (_Py_IsMainInterpreter(interp)) { _PyFloat_Init(); From webhook-mailer at python.org Thu Oct 28 13:06:25 2021 From: webhook-mailer at python.org (pablogsal) Date: Thu, 28 Oct 2021 17:06:25 -0000 Subject: [Python-checkins] bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270) Message-ID: https://github.com/python/cpython/commit/cdc7a5827754bec83970bb052d410d55f85b3fff commit: cdc7a5827754bec83970bb052d410d55f85b3fff branch: main author: Pablo Galindo Salgado committer: pablogsal date: 2021-10-28T18:06:15+01:00 summary: bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270) files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 4329cdd226e9c..8a19458ec72f4 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1048,7 +1048,7 @@ tok_nextc(struct tok_state *tok) #if defined(Py_DEBUG) if (Py_DebugFlag) { fprintf(stderr, "line[%d] = ", tok->lineno); - print_escape(stdout, tok->cur, tok->inp - tok->cur); + print_escape(stderr, tok->cur, tok->inp - tok->cur); fprintf(stderr, " tok->done = %d\n", tok->done); } #endif From webhook-mailer at python.org Thu Oct 28 14:22:10 2021 From: webhook-mailer at python.org (ned-deily) Date: Thu, 28 Oct 2021 18:22:10 -0000 Subject: [Python-checkins] bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) Message-ID: https://github.com/python/cpython/commit/be8318be05e1a874215fa75b8845ede74b2c69b6 commit: be8318be05e1a874215fa75b8845ede74b2c69b6 branch: main author: Ned Deily committer: ned-deily date: 2021-10-28T14:22:05-04:00 summary: bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) when using the Tk 8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the Tk project. files: A Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch A Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch new file mode 100644 index 0000000000000..1d06329fff3fb --- /dev/null +++ b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch @@ -0,0 +1,202 @@ +--- tk8.6.11/macosx/tkMacOSXDialog.c 2020-12-31 01:46:07.000000000 +0000 ++++ tk8.6.11-patched/macosx/tkMacOSXDialog.c 2021-10-28 15:13:03.000000000 +0000 +@@ -221,7 +221,7 @@ + returnCode: (NSModalResponse) returnCode + contextInfo: (void *) contextInfo + { +- FilePanelCallbackInfo *callbackInfo = contextInfo; ++ FilePanelCallbackInfo *callbackInfo = (FilePanelCallbackInfo *)contextInfo; + + if (returnCode == modalOK) { + Tcl_Obj *resultObj; +@@ -266,7 +266,7 @@ + - (void) tkAlertDidEnd: (NSAlert *) alert returnCode: (NSInteger) returnCode + contextInfo: (void *) contextInfo + { +- AlertCallbackInfo *callbackInfo = contextInfo; ++ AlertCallbackInfo *callbackInfo = (AlertCallbackInfo *)contextInfo; + + if (returnCode >= NSAlertFirstButtonReturn) { + Tcl_Obj *resultObj = Tcl_NewStringObj(alertButtonStrings[ +@@ -350,49 +350,41 @@ + FilePanelCallbackInfo *callbackInfo) + { + NSInteger modalReturnCode; ++ int OSVersion = [NSApp macOSVersion]; + +- if (parent && ![parent attachedSheet]) { +- [panel beginSheetModalForWindow:parent +- completionHandler:^(NSModalResponse returnCode) { +- [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; +- +- /* +- * The sheet has been prepared, so now we have to run it as a modal +- * window. Using [NSApp runModalForWindow:] on macOS 10.15 or later +- * generates warnings on stderr. But using [NSOpenPanel runModal] or +- * [NSSavePanel runModal] on 10.14 or earler does not cause the +- * completion handler to run when the panel is closed. +- */ ++ /* ++ * Use a sheet if -parent is specified (unless there is already a sheet). ++ */ + +- if ([NSApp macOSVersion] > 101400) { +- modalReturnCode = [panel runModal]; +- } else { ++ if (parent && ![parent attachedSheet]) { ++ if (OSVersion < 101500) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [NSApp runModalForWindow:panel]; +- } +- } else { +- +- /* +- * For the standalone file dialog, completion handlers do not work +- * at all on macOS 10.14 and earlier. +- */ +- +- if ([NSApp macOSVersion] > 101400) { +- [panel beginWithCompletionHandler:^(NSModalResponse returnCode) { ++ } else if (OSVersion < 110000) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { + [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [panel runModal]; + } else { ++ [parent beginSheet: panel completionHandler:nil]; + modalReturnCode = [panel runModal]; + [NSApp tkFilePanelDidEnd:panel +- returnCode:modalReturnCode +- contextInfo:callbackInfo ]; +- [panel close]; ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } ++ } else { ++ modalReturnCode = [panel runModal]; ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } + return callbackInfo->cmdObj ? modalOther : modalReturnCode; + } +@@ -422,7 +414,7 @@ + Tcl_Obj *const objv[]) /* Argument objects. */ + { + int result = TCL_ERROR; +- Tk_Window parent, tkwin = clientData; ++ Tk_Window parent, tkwin = (Tk_Window)clientData; + const char *title = NULL; + int i; + NSColor *color = nil, *initialColor = nil; +@@ -677,7 +669,7 @@ + int objc, /* Number of arguments. */ + Tcl_Obj *const objv[]) /* Argument objects. */ + { +- Tk_Window tkwin = clientData; ++ Tk_Window tkwin = (Tk_Window)clientData; + char *str; + int i, result = TCL_ERROR, haveParentOption = 0; + int index, len, multiple = 0; +@@ -1679,10 +1671,10 @@ + if (!fontchooserInterp) { + return; + } +- fcdPtr = Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); ++ fcdPtr = (FontchooserData *)Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); + switch (kind) { + case FontchooserClosed: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + TkSendVirtualEvent(fcdPtr->parent, "TkFontchooserVisibility", NULL); + fontchooserInterp = NULL; + } +@@ -1738,7 +1730,7 @@ + + switch(optionIndex) { + case FontchooserParent: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + resObj = Tcl_NewStringObj( + ((TkWindow *)fcdPtr->parent)->pathName, -1); + } else { +@@ -1801,7 +1793,7 @@ + Tcl_Obj *const objv[]) + { + Tk_Window tkwin = (Tk_Window)clientData; +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + int i, r = TCL_OK; + +@@ -1858,7 +1850,7 @@ + Tk_Window parent = Tk_NameToWindow(interp, + Tcl_GetString(objv[i+1]), tkwin); + +- if (parent == None) { ++ if (parent == NULL) { + return TCL_ERROR; + } + if (fcdPtr->parent) { +@@ -1885,7 +1877,7 @@ + fcdPtr->titleObj = NULL; + } + break; +- case FontchooserFont: ++ case FontchooserFont: { + Tcl_GetStringFromObj(objv[i+1], &len); + if (len) { + Tk_Font f = Tk_AllocFontFromObj(interp, tkwin, objv[i+1]); +@@ -1919,6 +1911,7 @@ + "TkFontchooserFontChanged", NULL); + } + break; ++ } + case FontchooserCmd: + if (fcdPtr->cmdObj) { + Tcl_DecrRefCount(fcdPtr->cmdObj); +@@ -1964,10 +1957,10 @@ + TCL_UNUSED(int), + TCL_UNUSED(Tcl_Obj *const *)) + { +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + +- if (fcdPtr->parent == None) { ++ if (fcdPtr->parent == NULL) { + fcdPtr->parent = (Tk_Window)clientData; + Tk_CreateEventHandler(fcdPtr->parent, StructureNotifyMask, + FontchooserParentEventHandler, fcdPtr); +@@ -2042,7 +2035,7 @@ + ClientData clientData, + XEvent *eventPtr) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (eventPtr->type == DestroyNotify) { + Tk_DeleteEventHandler(fcdPtr->parent, StructureNotifyMask, +@@ -2074,7 +2067,7 @@ + ClientData clientData, + Tcl_Interp *interp) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (fcdPtr->titleObj) { + Tcl_DecrRefCount(fcdPtr->titleObj); diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index a6d5c349c348b..f366968a71d91 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -269,7 +269,7 @@ def library_recipes(): tcl_checksum='8a4c004f48984a03a7747e9ba06e4da4' tk_checksum='c7ee71a2d05bba78dfffd76528dc17c6' - tk_patches = [ ] + tk_patches = ['bpo-44828-filedialog-crash-monterey.patch'] result.extend([ diff --git a/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst new file mode 100644 index 0000000000000..021d7e4d73782 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst @@ -0,0 +1,3 @@ +Avoid tkinter file dialog failure on macOS 12 Monterey when using the Tk +8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the +Tk project. From webhook-mailer at python.org Thu Oct 28 14:43:14 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Oct 2021 18:43:14 -0000 Subject: [Python-checkins] bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) Message-ID: https://github.com/python/cpython/commit/54579087c69f95531cbe7a97401c67f104a3e52f commit: 54579087c69f95531cbe7a97401c67f104a3e52f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-28T11:43:04-07:00 summary: bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) when using the Tk 8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the Tk project. (cherry picked from commit be8318be05e1a874215fa75b8845ede74b2c69b6) Co-authored-by: Ned Deily files: A Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch A Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch new file mode 100644 index 0000000000000..1d06329fff3fb --- /dev/null +++ b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch @@ -0,0 +1,202 @@ +--- tk8.6.11/macosx/tkMacOSXDialog.c 2020-12-31 01:46:07.000000000 +0000 ++++ tk8.6.11-patched/macosx/tkMacOSXDialog.c 2021-10-28 15:13:03.000000000 +0000 +@@ -221,7 +221,7 @@ + returnCode: (NSModalResponse) returnCode + contextInfo: (void *) contextInfo + { +- FilePanelCallbackInfo *callbackInfo = contextInfo; ++ FilePanelCallbackInfo *callbackInfo = (FilePanelCallbackInfo *)contextInfo; + + if (returnCode == modalOK) { + Tcl_Obj *resultObj; +@@ -266,7 +266,7 @@ + - (void) tkAlertDidEnd: (NSAlert *) alert returnCode: (NSInteger) returnCode + contextInfo: (void *) contextInfo + { +- AlertCallbackInfo *callbackInfo = contextInfo; ++ AlertCallbackInfo *callbackInfo = (AlertCallbackInfo *)contextInfo; + + if (returnCode >= NSAlertFirstButtonReturn) { + Tcl_Obj *resultObj = Tcl_NewStringObj(alertButtonStrings[ +@@ -350,49 +350,41 @@ + FilePanelCallbackInfo *callbackInfo) + { + NSInteger modalReturnCode; ++ int OSVersion = [NSApp macOSVersion]; + +- if (parent && ![parent attachedSheet]) { +- [panel beginSheetModalForWindow:parent +- completionHandler:^(NSModalResponse returnCode) { +- [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; +- +- /* +- * The sheet has been prepared, so now we have to run it as a modal +- * window. Using [NSApp runModalForWindow:] on macOS 10.15 or later +- * generates warnings on stderr. But using [NSOpenPanel runModal] or +- * [NSSavePanel runModal] on 10.14 or earler does not cause the +- * completion handler to run when the panel is closed. +- */ ++ /* ++ * Use a sheet if -parent is specified (unless there is already a sheet). ++ */ + +- if ([NSApp macOSVersion] > 101400) { +- modalReturnCode = [panel runModal]; +- } else { ++ if (parent && ![parent attachedSheet]) { ++ if (OSVersion < 101500) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [NSApp runModalForWindow:panel]; +- } +- } else { +- +- /* +- * For the standalone file dialog, completion handlers do not work +- * at all on macOS 10.14 and earlier. +- */ +- +- if ([NSApp macOSVersion] > 101400) { +- [panel beginWithCompletionHandler:^(NSModalResponse returnCode) { ++ } else if (OSVersion < 110000) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { + [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [panel runModal]; + } else { ++ [parent beginSheet: panel completionHandler:nil]; + modalReturnCode = [panel runModal]; + [NSApp tkFilePanelDidEnd:panel +- returnCode:modalReturnCode +- contextInfo:callbackInfo ]; +- [panel close]; ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } ++ } else { ++ modalReturnCode = [panel runModal]; ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } + return callbackInfo->cmdObj ? modalOther : modalReturnCode; + } +@@ -422,7 +414,7 @@ + Tcl_Obj *const objv[]) /* Argument objects. */ + { + int result = TCL_ERROR; +- Tk_Window parent, tkwin = clientData; ++ Tk_Window parent, tkwin = (Tk_Window)clientData; + const char *title = NULL; + int i; + NSColor *color = nil, *initialColor = nil; +@@ -677,7 +669,7 @@ + int objc, /* Number of arguments. */ + Tcl_Obj *const objv[]) /* Argument objects. */ + { +- Tk_Window tkwin = clientData; ++ Tk_Window tkwin = (Tk_Window)clientData; + char *str; + int i, result = TCL_ERROR, haveParentOption = 0; + int index, len, multiple = 0; +@@ -1679,10 +1671,10 @@ + if (!fontchooserInterp) { + return; + } +- fcdPtr = Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); ++ fcdPtr = (FontchooserData *)Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); + switch (kind) { + case FontchooserClosed: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + TkSendVirtualEvent(fcdPtr->parent, "TkFontchooserVisibility", NULL); + fontchooserInterp = NULL; + } +@@ -1738,7 +1730,7 @@ + + switch(optionIndex) { + case FontchooserParent: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + resObj = Tcl_NewStringObj( + ((TkWindow *)fcdPtr->parent)->pathName, -1); + } else { +@@ -1801,7 +1793,7 @@ + Tcl_Obj *const objv[]) + { + Tk_Window tkwin = (Tk_Window)clientData; +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + int i, r = TCL_OK; + +@@ -1858,7 +1850,7 @@ + Tk_Window parent = Tk_NameToWindow(interp, + Tcl_GetString(objv[i+1]), tkwin); + +- if (parent == None) { ++ if (parent == NULL) { + return TCL_ERROR; + } + if (fcdPtr->parent) { +@@ -1885,7 +1877,7 @@ + fcdPtr->titleObj = NULL; + } + break; +- case FontchooserFont: ++ case FontchooserFont: { + Tcl_GetStringFromObj(objv[i+1], &len); + if (len) { + Tk_Font f = Tk_AllocFontFromObj(interp, tkwin, objv[i+1]); +@@ -1919,6 +1911,7 @@ + "TkFontchooserFontChanged", NULL); + } + break; ++ } + case FontchooserCmd: + if (fcdPtr->cmdObj) { + Tcl_DecrRefCount(fcdPtr->cmdObj); +@@ -1964,10 +1957,10 @@ + TCL_UNUSED(int), + TCL_UNUSED(Tcl_Obj *const *)) + { +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + +- if (fcdPtr->parent == None) { ++ if (fcdPtr->parent == NULL) { + fcdPtr->parent = (Tk_Window)clientData; + Tk_CreateEventHandler(fcdPtr->parent, StructureNotifyMask, + FontchooserParentEventHandler, fcdPtr); +@@ -2042,7 +2035,7 @@ + ClientData clientData, + XEvent *eventPtr) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (eventPtr->type == DestroyNotify) { + Tk_DeleteEventHandler(fcdPtr->parent, StructureNotifyMask, +@@ -2074,7 +2067,7 @@ + ClientData clientData, + Tcl_Interp *interp) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (fcdPtr->titleObj) { + Tcl_DecrRefCount(fcdPtr->titleObj); diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index 51c8523a8880b..86d31688da75c 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -269,7 +269,7 @@ def library_recipes(): tcl_checksum='8a4c004f48984a03a7747e9ba06e4da4' tk_checksum='c7ee71a2d05bba78dfffd76528dc17c6' - tk_patches = [ ] + tk_patches = ['bpo-44828-filedialog-crash-monterey.patch'] result.extend([ diff --git a/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst new file mode 100644 index 0000000000000..021d7e4d73782 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst @@ -0,0 +1,3 @@ +Avoid tkinter file dialog failure on macOS 12 Monterey when using the Tk +8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the +Tk project. From webhook-mailer at python.org Thu Oct 28 14:48:06 2021 From: webhook-mailer at python.org (miss-islington) Date: Thu, 28 Oct 2021 18:48:06 -0000 Subject: [Python-checkins] bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) Message-ID: https://github.com/python/cpython/commit/8e5e74e3049875e9d834fe4408263676fe21e890 commit: 8e5e74e3049875e9d834fe4408263676fe21e890 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-28T11:47:53-07:00 summary: bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) when using the Tk 8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the Tk project. (cherry picked from commit be8318be05e1a874215fa75b8845ede74b2c69b6) Co-authored-by: Ned Deily files: A Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch A Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch new file mode 100644 index 0000000000000..1d06329fff3fb --- /dev/null +++ b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch @@ -0,0 +1,202 @@ +--- tk8.6.11/macosx/tkMacOSXDialog.c 2020-12-31 01:46:07.000000000 +0000 ++++ tk8.6.11-patched/macosx/tkMacOSXDialog.c 2021-10-28 15:13:03.000000000 +0000 +@@ -221,7 +221,7 @@ + returnCode: (NSModalResponse) returnCode + contextInfo: (void *) contextInfo + { +- FilePanelCallbackInfo *callbackInfo = contextInfo; ++ FilePanelCallbackInfo *callbackInfo = (FilePanelCallbackInfo *)contextInfo; + + if (returnCode == modalOK) { + Tcl_Obj *resultObj; +@@ -266,7 +266,7 @@ + - (void) tkAlertDidEnd: (NSAlert *) alert returnCode: (NSInteger) returnCode + contextInfo: (void *) contextInfo + { +- AlertCallbackInfo *callbackInfo = contextInfo; ++ AlertCallbackInfo *callbackInfo = (AlertCallbackInfo *)contextInfo; + + if (returnCode >= NSAlertFirstButtonReturn) { + Tcl_Obj *resultObj = Tcl_NewStringObj(alertButtonStrings[ +@@ -350,49 +350,41 @@ + FilePanelCallbackInfo *callbackInfo) + { + NSInteger modalReturnCode; ++ int OSVersion = [NSApp macOSVersion]; + +- if (parent && ![parent attachedSheet]) { +- [panel beginSheetModalForWindow:parent +- completionHandler:^(NSModalResponse returnCode) { +- [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; +- +- /* +- * The sheet has been prepared, so now we have to run it as a modal +- * window. Using [NSApp runModalForWindow:] on macOS 10.15 or later +- * generates warnings on stderr. But using [NSOpenPanel runModal] or +- * [NSSavePanel runModal] on 10.14 or earler does not cause the +- * completion handler to run when the panel is closed. +- */ ++ /* ++ * Use a sheet if -parent is specified (unless there is already a sheet). ++ */ + +- if ([NSApp macOSVersion] > 101400) { +- modalReturnCode = [panel runModal]; +- } else { ++ if (parent && ![parent attachedSheet]) { ++ if (OSVersion < 101500) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [NSApp runModalForWindow:panel]; +- } +- } else { +- +- /* +- * For the standalone file dialog, completion handlers do not work +- * at all on macOS 10.14 and earlier. +- */ +- +- if ([NSApp macOSVersion] > 101400) { +- [panel beginWithCompletionHandler:^(NSModalResponse returnCode) { ++ } else if (OSVersion < 110000) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { + [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [panel runModal]; + } else { ++ [parent beginSheet: panel completionHandler:nil]; + modalReturnCode = [panel runModal]; + [NSApp tkFilePanelDidEnd:panel +- returnCode:modalReturnCode +- contextInfo:callbackInfo ]; +- [panel close]; ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } ++ } else { ++ modalReturnCode = [panel runModal]; ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } + return callbackInfo->cmdObj ? modalOther : modalReturnCode; + } +@@ -422,7 +414,7 @@ + Tcl_Obj *const objv[]) /* Argument objects. */ + { + int result = TCL_ERROR; +- Tk_Window parent, tkwin = clientData; ++ Tk_Window parent, tkwin = (Tk_Window)clientData; + const char *title = NULL; + int i; + NSColor *color = nil, *initialColor = nil; +@@ -677,7 +669,7 @@ + int objc, /* Number of arguments. */ + Tcl_Obj *const objv[]) /* Argument objects. */ + { +- Tk_Window tkwin = clientData; ++ Tk_Window tkwin = (Tk_Window)clientData; + char *str; + int i, result = TCL_ERROR, haveParentOption = 0; + int index, len, multiple = 0; +@@ -1679,10 +1671,10 @@ + if (!fontchooserInterp) { + return; + } +- fcdPtr = Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); ++ fcdPtr = (FontchooserData *)Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); + switch (kind) { + case FontchooserClosed: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + TkSendVirtualEvent(fcdPtr->parent, "TkFontchooserVisibility", NULL); + fontchooserInterp = NULL; + } +@@ -1738,7 +1730,7 @@ + + switch(optionIndex) { + case FontchooserParent: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + resObj = Tcl_NewStringObj( + ((TkWindow *)fcdPtr->parent)->pathName, -1); + } else { +@@ -1801,7 +1793,7 @@ + Tcl_Obj *const objv[]) + { + Tk_Window tkwin = (Tk_Window)clientData; +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + int i, r = TCL_OK; + +@@ -1858,7 +1850,7 @@ + Tk_Window parent = Tk_NameToWindow(interp, + Tcl_GetString(objv[i+1]), tkwin); + +- if (parent == None) { ++ if (parent == NULL) { + return TCL_ERROR; + } + if (fcdPtr->parent) { +@@ -1885,7 +1877,7 @@ + fcdPtr->titleObj = NULL; + } + break; +- case FontchooserFont: ++ case FontchooserFont: { + Tcl_GetStringFromObj(objv[i+1], &len); + if (len) { + Tk_Font f = Tk_AllocFontFromObj(interp, tkwin, objv[i+1]); +@@ -1919,6 +1911,7 @@ + "TkFontchooserFontChanged", NULL); + } + break; ++ } + case FontchooserCmd: + if (fcdPtr->cmdObj) { + Tcl_DecrRefCount(fcdPtr->cmdObj); +@@ -1964,10 +1957,10 @@ + TCL_UNUSED(int), + TCL_UNUSED(Tcl_Obj *const *)) + { +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + +- if (fcdPtr->parent == None) { ++ if (fcdPtr->parent == NULL) { + fcdPtr->parent = (Tk_Window)clientData; + Tk_CreateEventHandler(fcdPtr->parent, StructureNotifyMask, + FontchooserParentEventHandler, fcdPtr); +@@ -2042,7 +2035,7 @@ + ClientData clientData, + XEvent *eventPtr) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (eventPtr->type == DestroyNotify) { + Tk_DeleteEventHandler(fcdPtr->parent, StructureNotifyMask, +@@ -2074,7 +2067,7 @@ + ClientData clientData, + Tcl_Interp *interp) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (fcdPtr->titleObj) { + Tcl_DecrRefCount(fcdPtr->titleObj); diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index bd1c4d4fe6d8d..10112fe330061 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -267,7 +267,7 @@ def library_recipes(): tcl_checksum='8a4c004f48984a03a7747e9ba06e4da4' tk_checksum='c7ee71a2d05bba78dfffd76528dc17c6' - tk_patches = [ ] + tk_patches = ['bpo-44828-filedialog-crash-monterey.patch'] result.extend([ diff --git a/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst new file mode 100644 index 0000000000000..021d7e4d73782 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst @@ -0,0 +1,3 @@ +Avoid tkinter file dialog failure on macOS 12 Monterey when using the Tk +8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the +Tk project. From webhook-mailer at python.org Thu Oct 28 15:08:47 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:08:47 -0000 Subject: [Python-checkins] [3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) (GH-29245) Message-ID: https://github.com/python/cpython/commit/f2407144347fafcd69c2ade41b5d9c3fb07b59ef commit: f2407144347fafcd69c2ade41b5d9c3fb07b59ef branch: 3.8 author: Ned Deily committer: ambv date: 2021-10-28T21:08:42+02:00 summary: [3.9] bpo-45618: Fix documentation build by pinning Docutils version to 0.17.1 (GH-29230) (GH-29241) (GH-29245) Co-authored-by: Maciej Olko Co-authored-by: Erlend Egeberg Aasland files: M .azure-pipelines/docs-steps.yml M Doc/requirements.txt diff --git a/.azure-pipelines/docs-steps.yml b/.azure-pipelines/docs-steps.yml index 492e4e34bb2da..ae75339b57a1d 100644 --- a/.azure-pipelines/docs-steps.yml +++ b/.azure-pipelines/docs-steps.yml @@ -12,11 +12,12 @@ steps: inputs: versionSpec: '>=3.6' -- script: python -m pip install sphinx==1.8.2 blurb python-docs-theme +- script: python -m pip install -r requirements.txt + workingDirectory: '$(build.sourcesDirectory)/Doc' displayName: 'Install build dependencies' - ${{ if ne(parameters.latex, 'true') }}: - - script: make check suspicious html PYTHON=python + - script: make check html PYTHON=python workingDirectory: '$(build.sourcesDirectory)/Doc' displayName: 'Build documentation' @@ -31,7 +32,7 @@ steps: - ${{ if eq(parameters.upload, 'true') }}: - task: PublishBuildArtifacts at 1 displayName: 'Publish docs' - + inputs: PathToPublish: '$(build.sourcesDirectory)/Doc/build' ArtifactName: docs diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 47b78eeac817e..cb21ed20397b2 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -4,6 +4,10 @@ # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. sphinx==2.4.4 +# Docutils version is pinned to a version compatible with Sphinx +# version 2.4.4. It can be removed after bumping Sphinx version to at +# least 3.5.4. +docutils==0.17.1 blurb From webhook-mailer at python.org Thu Oct 28 15:10:20 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:10:20 -0000 Subject: [Python-checkins] bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) (GH-29279) Message-ID: https://github.com/python/cpython/commit/f19c1a115f782036edac306de0f3f9968c1e1fd6 commit: f19c1a115f782036edac306de0f3f9968c1e1fd6 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T21:10:15+02:00 summary: bpo-44828: Avoid tkinter file dialog failure on macOS 12 Monterey (GH-29276) (GH-29279) when using the Tk 8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the Tk project. (cherry picked from commit be8318be05e1a874215fa75b8845ede74b2c69b6) Co-authored-by: Ned Deily files: A Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch A Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst M Mac/BuildScript/build-installer.py diff --git a/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch new file mode 100644 index 0000000000000..1d06329fff3fb --- /dev/null +++ b/Mac/BuildScript/bpo-44828-filedialog-crash-monterey.patch @@ -0,0 +1,202 @@ +--- tk8.6.11/macosx/tkMacOSXDialog.c 2020-12-31 01:46:07.000000000 +0000 ++++ tk8.6.11-patched/macosx/tkMacOSXDialog.c 2021-10-28 15:13:03.000000000 +0000 +@@ -221,7 +221,7 @@ + returnCode: (NSModalResponse) returnCode + contextInfo: (void *) contextInfo + { +- FilePanelCallbackInfo *callbackInfo = contextInfo; ++ FilePanelCallbackInfo *callbackInfo = (FilePanelCallbackInfo *)contextInfo; + + if (returnCode == modalOK) { + Tcl_Obj *resultObj; +@@ -266,7 +266,7 @@ + - (void) tkAlertDidEnd: (NSAlert *) alert returnCode: (NSInteger) returnCode + contextInfo: (void *) contextInfo + { +- AlertCallbackInfo *callbackInfo = contextInfo; ++ AlertCallbackInfo *callbackInfo = (AlertCallbackInfo *)contextInfo; + + if (returnCode >= NSAlertFirstButtonReturn) { + Tcl_Obj *resultObj = Tcl_NewStringObj(alertButtonStrings[ +@@ -350,49 +350,41 @@ + FilePanelCallbackInfo *callbackInfo) + { + NSInteger modalReturnCode; ++ int OSVersion = [NSApp macOSVersion]; + +- if (parent && ![parent attachedSheet]) { +- [panel beginSheetModalForWindow:parent +- completionHandler:^(NSModalResponse returnCode) { +- [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; +- +- /* +- * The sheet has been prepared, so now we have to run it as a modal +- * window. Using [NSApp runModalForWindow:] on macOS 10.15 or later +- * generates warnings on stderr. But using [NSOpenPanel runModal] or +- * [NSSavePanel runModal] on 10.14 or earler does not cause the +- * completion handler to run when the panel is closed. +- */ ++ /* ++ * Use a sheet if -parent is specified (unless there is already a sheet). ++ */ + +- if ([NSApp macOSVersion] > 101400) { +- modalReturnCode = [panel runModal]; +- } else { ++ if (parent && ![parent attachedSheet]) { ++ if (OSVersion < 101500) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [NSApp runModalForWindow:panel]; +- } +- } else { +- +- /* +- * For the standalone file dialog, completion handlers do not work +- * at all on macOS 10.14 and earlier. +- */ +- +- if ([NSApp macOSVersion] > 101400) { +- [panel beginWithCompletionHandler:^(NSModalResponse returnCode) { ++ } else if (OSVersion < 110000) { ++ [panel beginSheetModalForWindow:parent ++ completionHandler:^(NSModalResponse returnCode) { + [NSApp tkFilePanelDidEnd:panel +- returnCode:returnCode +- contextInfo:callbackInfo ]; +- }]; ++ returnCode:returnCode ++ contextInfo:callbackInfo ]; ++ }]; + modalReturnCode = [panel runModal]; + } else { ++ [parent beginSheet: panel completionHandler:nil]; + modalReturnCode = [panel runModal]; + [NSApp tkFilePanelDidEnd:panel +- returnCode:modalReturnCode +- contextInfo:callbackInfo ]; +- [panel close]; ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } ++ } else { ++ modalReturnCode = [panel runModal]; ++ [NSApp tkFilePanelDidEnd:panel ++ returnCode:modalReturnCode ++ contextInfo:callbackInfo ]; + } + return callbackInfo->cmdObj ? modalOther : modalReturnCode; + } +@@ -422,7 +414,7 @@ + Tcl_Obj *const objv[]) /* Argument objects. */ + { + int result = TCL_ERROR; +- Tk_Window parent, tkwin = clientData; ++ Tk_Window parent, tkwin = (Tk_Window)clientData; + const char *title = NULL; + int i; + NSColor *color = nil, *initialColor = nil; +@@ -677,7 +669,7 @@ + int objc, /* Number of arguments. */ + Tcl_Obj *const objv[]) /* Argument objects. */ + { +- Tk_Window tkwin = clientData; ++ Tk_Window tkwin = (Tk_Window)clientData; + char *str; + int i, result = TCL_ERROR, haveParentOption = 0; + int index, len, multiple = 0; +@@ -1679,10 +1671,10 @@ + if (!fontchooserInterp) { + return; + } +- fcdPtr = Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); ++ fcdPtr = (FontchooserData *)Tcl_GetAssocData(fontchooserInterp, "::tk::fontchooser", NULL); + switch (kind) { + case FontchooserClosed: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + TkSendVirtualEvent(fcdPtr->parent, "TkFontchooserVisibility", NULL); + fontchooserInterp = NULL; + } +@@ -1738,7 +1730,7 @@ + + switch(optionIndex) { + case FontchooserParent: +- if (fcdPtr->parent != None) { ++ if (fcdPtr->parent != NULL) { + resObj = Tcl_NewStringObj( + ((TkWindow *)fcdPtr->parent)->pathName, -1); + } else { +@@ -1801,7 +1793,7 @@ + Tcl_Obj *const objv[]) + { + Tk_Window tkwin = (Tk_Window)clientData; +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + int i, r = TCL_OK; + +@@ -1858,7 +1850,7 @@ + Tk_Window parent = Tk_NameToWindow(interp, + Tcl_GetString(objv[i+1]), tkwin); + +- if (parent == None) { ++ if (parent == NULL) { + return TCL_ERROR; + } + if (fcdPtr->parent) { +@@ -1885,7 +1877,7 @@ + fcdPtr->titleObj = NULL; + } + break; +- case FontchooserFont: ++ case FontchooserFont: { + Tcl_GetStringFromObj(objv[i+1], &len); + if (len) { + Tk_Font f = Tk_AllocFontFromObj(interp, tkwin, objv[i+1]); +@@ -1919,6 +1911,7 @@ + "TkFontchooserFontChanged", NULL); + } + break; ++ } + case FontchooserCmd: + if (fcdPtr->cmdObj) { + Tcl_DecrRefCount(fcdPtr->cmdObj); +@@ -1964,10 +1957,10 @@ + TCL_UNUSED(int), + TCL_UNUSED(Tcl_Obj *const *)) + { +- FontchooserData *fcdPtr = Tcl_GetAssocData(interp, "::tk::fontchooser", ++ FontchooserData *fcdPtr = (FontchooserData *)Tcl_GetAssocData(interp, "::tk::fontchooser", + NULL); + +- if (fcdPtr->parent == None) { ++ if (fcdPtr->parent == NULL) { + fcdPtr->parent = (Tk_Window)clientData; + Tk_CreateEventHandler(fcdPtr->parent, StructureNotifyMask, + FontchooserParentEventHandler, fcdPtr); +@@ -2042,7 +2035,7 @@ + ClientData clientData, + XEvent *eventPtr) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (eventPtr->type == DestroyNotify) { + Tk_DeleteEventHandler(fcdPtr->parent, StructureNotifyMask, +@@ -2074,7 +2067,7 @@ + ClientData clientData, + Tcl_Interp *interp) + { +- FontchooserData *fcdPtr = clientData; ++ FontchooserData *fcdPtr = (FontchooserData *)clientData; + + if (fcdPtr->titleObj) { + Tcl_DecrRefCount(fcdPtr->titleObj); diff --git a/Mac/BuildScript/build-installer.py b/Mac/BuildScript/build-installer.py index ec2be50883015..db25eb890625c 100755 --- a/Mac/BuildScript/build-installer.py +++ b/Mac/BuildScript/build-installer.py @@ -265,7 +265,7 @@ def library_recipes(): tcl_checksum='8a4c004f48984a03a7747e9ba06e4da4' tk_checksum='c7ee71a2d05bba78dfffd76528dc17c6' - tk_patches = [ ] + tk_patches = ['bpo-44828-filedialog-crash-monterey.patch'] result.extend([ diff --git a/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst new file mode 100644 index 0000000000000..021d7e4d73782 --- /dev/null +++ b/Misc/NEWS.d/next/macOS/2021-10-25-02-02-21.bpo-44828.XBdXlJ.rst @@ -0,0 +1,3 @@ +Avoid tkinter file dialog failure on macOS 12 Monterey when using the Tk +8.6.11 provided by python.org macOS installers. Patch by Marc Culler of the +Tk project. From webhook-mailer at python.org Thu Oct 28 15:23:05 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:23:05 -0000 Subject: [Python-checkins] [3.10] bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) (GH-29249) Message-ID: https://github.com/python/cpython/commit/9e0012116ac9e8d26bf19ef8741deeecf2b6f72b commit: 9e0012116ac9e8d26bf19ef8741deeecf2b6f72b branch: 3.10 author: Sam Gross committer: ambv date: 2021-10-28T21:22:57+02:00 summary: [3.10] bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) (GH-29249) Objects that support garbage collection ("container" objects) should call PyObject_GC_UnTrack() from their destructors before clearing any fields which may point to other "container" objects. (cherry picked from commit 35e1ff38ee67ee543d9fcb268c3552c5397f9b3f) Co-authored-by: Sam Gross files: M Doc/c-api/gcsupport.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes.rst diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index 4bd2fb3837a1a..8c90d1e8991c1 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -33,6 +33,14 @@ Constructors for container types must conform to two rules: #. Once all the fields which may contain references to other containers are initialized, it must call :c:func:`PyObject_GC_Track`. +Similarly, the deallocator for the object must conform to a similar pair of +rules: + +#. Before fields which refer to other containers are invalidated, + :c:func:`PyObject_GC_UnTrack` must be called. + +#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. + .. warning:: If a type adds the Py_TPFLAGS_HAVE_GC, then it *must* implement at least a :c:member:`~PyTypeObject.tp_traverse` handler or explicitly use one @@ -100,14 +108,6 @@ Constructors for container types must conform to two rules: .. versionadded:: 3.9 -Similarly, the deallocator for the object must conform to a similar pair of -rules: - -#. Before fields which refer to other containers are invalidated, - :c:func:`PyObject_GC_UnTrack` must be called. - -#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. - .. c:function:: void PyObject_GC_Del(void *op) diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index b642835a480db..faef6ab5fed50 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -668,6 +668,18 @@ and :c:type:`PyType_Type` effectively act as defaults.) :c:func:`PyObject_GC_Del` if the instance was allocated using :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. + If the type supports garbage collection (has the :const:`Py_TPFLAGS_HAVE_GC` + flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack` + before clearing any member fields. + + .. code-block:: c + + static void foo_dealloc(foo_object *self) { + PyObject_GC_UnTrack(self); + Py_CLEAR(self->ref); + Py_TYPE(self)->tp_free((PyObject *)self); + } + Finally, if the type is heap allocated (:const:`Py_TPFLAGS_HEAPTYPE`), the deallocator should decrement the reference count for its type object after calling the type deallocator. In order to avoid dangling pointers, the diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index 6e17897ed2c80..23ec8bce8c5ac 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -73,7 +73,19 @@ function:: newdatatype_dealloc(newdatatypeobject *obj) { free(obj->obj_UnderlyingDatatypePtr); - Py_TYPE(obj)->tp_free(obj); + Py_TYPE(obj)->tp_free((PyObject *)obj); + } + +If your type supports garbage collection, the destructor should call +:c:func:`PyObject_GC_UnTrack` before clearing any member fields:: + + static void + newdatatype_dealloc(newdatatypeobject *obj) + { + PyObject_GC_UnTrack(obj); + Py_CLEAR(obj->other_obj); + ... + Py_TYPE(obj)->tp_free((PyObject *)obj); } .. index:: From webhook-mailer at python.org Thu Oct 28 15:23:23 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:23:23 -0000 Subject: [Python-checkins] bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) (GH-29248) Message-ID: https://github.com/python/cpython/commit/193504acf3bfb7cff1edf7f568c2405b857fa1f7 commit: 193504acf3bfb7cff1edf7f568c2405b857fa1f7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T21:23:19+02:00 summary: bpo-28737: Document when tp_dealloc should call PyObject_GC_UnTrack() (GH-29246) (GH-29248) Objects that support garbage collection ("container" objects) should call PyObject_GC_UnTrack() from their destructors before clearing any fields which may point to other "container" objects. (cherry picked from commit 35e1ff38ee67ee543d9fcb268c3552c5397f9b3f) Co-authored-by: Sam Gross files: M Doc/c-api/gcsupport.rst M Doc/c-api/typeobj.rst M Doc/extending/newtypes.rst diff --git a/Doc/c-api/gcsupport.rst b/Doc/c-api/gcsupport.rst index f821b45090c47..b0d62d8cf830a 100644 --- a/Doc/c-api/gcsupport.rst +++ b/Doc/c-api/gcsupport.rst @@ -33,6 +33,14 @@ Constructors for container types must conform to two rules: #. Once all the fields which may contain references to other containers are initialized, it must call :c:func:`PyObject_GC_Track`. +Similarly, the deallocator for the object must conform to a similar pair of +rules: + +#. Before fields which refer to other containers are invalidated, + :c:func:`PyObject_GC_UnTrack` must be called. + +#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. + .. warning:: If a type adds the Py_TPFLAGS_HAVE_GC, then it *must* implement at least a :c:member:`~PyTypeObject.tp_traverse` handler or explicitly use one @@ -100,14 +108,6 @@ Constructors for container types must conform to two rules: .. versionadded:: 3.9 -Similarly, the deallocator for the object must conform to a similar pair of -rules: - -#. Before fields which refer to other containers are invalidated, - :c:func:`PyObject_GC_UnTrack` must be called. - -#. The object's memory must be deallocated using :c:func:`PyObject_GC_Del`. - .. c:function:: void PyObject_GC_Del(void *op) diff --git a/Doc/c-api/typeobj.rst b/Doc/c-api/typeobj.rst index 661e87f68982d..a800616730b9d 100644 --- a/Doc/c-api/typeobj.rst +++ b/Doc/c-api/typeobj.rst @@ -657,6 +657,18 @@ and :c:type:`PyType_Type` effectively act as defaults.) :c:func:`PyObject_GC_Del` if the instance was allocated using :c:func:`PyObject_GC_New` or :c:func:`PyObject_GC_NewVar`. + If the type supports garbage collection (has the :const:`Py_TPFLAGS_HAVE_GC` + flag bit set), the destructor should call :c:func:`PyObject_GC_UnTrack` + before clearing any member fields. + + .. code-block:: c + + static void foo_dealloc(foo_object *self) { + PyObject_GC_UnTrack(self); + Py_CLEAR(self->ref); + Py_TYPE(self)->tp_free((PyObject *)self); + } + Finally, if the type is heap allocated (:const:`Py_TPFLAGS_HEAPTYPE`), the deallocator should decrement the reference count for its type object after calling the type deallocator. In order to avoid dangling pointers, the diff --git a/Doc/extending/newtypes.rst b/Doc/extending/newtypes.rst index d9023709fddc8..638b00a4c0f68 100644 --- a/Doc/extending/newtypes.rst +++ b/Doc/extending/newtypes.rst @@ -73,7 +73,19 @@ function:: newdatatype_dealloc(newdatatypeobject *obj) { free(obj->obj_UnderlyingDatatypePtr); - Py_TYPE(obj)->tp_free(obj); + Py_TYPE(obj)->tp_free((PyObject *)obj); + } + +If your type supports garbage collection, the destructor should call +:c:func:`PyObject_GC_UnTrack` before clearing any member fields:: + + static void + newdatatype_dealloc(newdatatypeobject *obj) + { + PyObject_GC_UnTrack(obj); + Py_CLEAR(obj->other_obj); + ... + Py_TYPE(obj)->tp_free((PyObject *)obj); } .. index:: From webhook-mailer at python.org Thu Oct 28 15:35:55 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:35:55 -0000 Subject: [Python-checkins] bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) Message-ID: https://github.com/python/cpython/commit/88d8a1a340fb09c54d47f354f5fd7d4fbc5f0c78 commit: 88d8a1a340fb09c54d47f354f5fd7d4fbc5f0c78 branch: main author: Erlend Egeberg Aasland committer: ambv date: 2021-10-28T21:35:51+02:00 summary: bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index eaea7ae390b97..fe1b64ade9561 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -117,6 +117,24 @@ Module functions and constants ------------------------------ +.. data:: apilevel + + String constant stating the supported DB-API level. Required by the DB-API. + Hard-coded to ``"2.0"``. + +.. data:: paramstyle + + String constant stating the type of parameter marker formatting expected by + the :mod:`sqlite3` module. Required by the DB-API. Hard-coded to + ``"qmark"``. + + .. note:: + + The :mod:`sqlite3` module supports both ``qmark`` and ``numeric`` DB-API + parameter styles, because that is what the underlying SQLite library + supports. However, the DB-API does not allow multiple values for + the ``paramstyle`` attribute. + .. data:: version The version number of this module, as a string. This is not the version of @@ -139,6 +157,26 @@ Module functions and constants The version number of the run-time SQLite library, as a tuple of integers. +.. data:: threadsafety + + Integer constant required by the DB-API, stating the level of thread safety + the :mod:`sqlite3` module supports. Currently hard-coded to ``1``, meaning + *"Threads may share the module, but not connections."* However, this may not + always be true. You can check the underlying SQLite library's compile-time + threaded mode using the following query:: + + import sqlite3 + con = sqlite3.connect(":memory:") + con.execute(""" + select * from pragma_compile_options + where compile_options like 'THREADSAFE=%' + """).fetchall() + + Note that the `SQLITE_THREADSAFE levels + `_ do not match the DB-API 2.0 + ``threadsafety`` levels. + + .. data:: PARSE_DECLTYPES This constant is meant to be used with the *detect_types* parameter of the @@ -710,6 +748,14 @@ Cursor Objects The cursor will be unusable from this point forward; a :exc:`ProgrammingError` exception will be raised if any operation is attempted with the cursor. + .. method:: setinputsizes(sizes) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + + .. method:: setoutputsize(size [, column]) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + .. attribute:: rowcount Although the :class:`Cursor` class of the :mod:`sqlite3` module implements this From webhook-mailer at python.org Thu Oct 28 15:38:19 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:38:19 -0000 Subject: [Python-checkins] bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) Message-ID: https://github.com/python/cpython/commit/1fb968c07a76fb2d1ec8c14a0026f1d15828f4a5 commit: 1fb968c07a76fb2d1ec8c14a0026f1d15828f4a5 branch: main author: Nikita Sobolev committer: ambv date: 2021-10-28T21:38:14+02:00 summary: bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst M Doc/library/multiprocessing.rst diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index f3d725b822529..7a1a285255ff7 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2636,12 +2636,13 @@ handler type) for messages from different processes to get mixed up. inherited. .. currentmodule:: multiprocessing -.. function:: log_to_stderr() +.. function:: log_to_stderr(level=None) This function performs a call to :func:`get_logger` but in addition to returning the logger created by get_logger, it adds a handler which sends output to :data:`sys.stderr` using format ``'[%(levelname)s/%(processName)s] %(message)s'``. + You can modify ``levelname`` of the logger by passing a ``level`` argument. Below is an example session with logging turned on:: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst new file mode 100644 index 0000000000000..9da9cca7bf1ea --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst @@ -0,0 +1 @@ +Add ``level`` argument to ``multiprocessing.log_to_stderr`` function docs. \ No newline at end of file From webhook-mailer at python.org Thu Oct 28 15:39:32 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:39:32 -0000 Subject: [Python-checkins] Fixed typos (GH-29211) Message-ID: https://github.com/python/cpython/commit/cd30613165add99c5d0c4592d1a3f4243f09797b commit: cd30613165add99c5d0c4592d1a3f4243f09797b branch: main author: MalikIdreesHasanKhan committer: ambv date: 2021-10-28T21:39:27+02:00 summary: Fixed typos (GH-29211) files: M Misc/HISTORY diff --git a/Misc/HISTORY b/Misc/HISTORY index 805acf4bebcad..570638869f92e 100644 --- a/Misc/HISTORY +++ b/Misc/HISTORY @@ -13756,7 +13756,7 @@ Library processing instruction. Patch by Neil Muller. - Issue #6233: ElementTree failed converting unicode characters to XML entities - when they could't be represented in the requested output encoding. Patch by + when they couldn't be represented in the requested output encoding. Patch by Jerry Chen. - Issue #6003: add an argument to ``zipfile.Zipfile.writestr`` to specify the @@ -20085,7 +20085,7 @@ Windows - Boosted the stack reservation for python.exe and pythonw.exe from the default 1MB to 2MB. Stack frames under VC 7.1 for 2.4 are enough - bigger than under VC 6.0 for 2.3.4 that deeply recursive progams + bigger than under VC 6.0 for 2.3.4 that deeply recursive programs within the default sys.getrecursionlimit() default value of 1000 were able to suffer undetected C stack overflows. The standard test program test_compiler was one such program. If a Python process on Windows @@ -27197,7 +27197,7 @@ Wed Apr 7 20:23:17 1999 Guido van Rossum """ Mostly I just re-added the SMTPRecipientsRefused exception - (the exeption object now has the appropriate info in it ) [Per had + (the exception object now has the appropriate info in it ) [Per had removed this in his patch --GvR] and tweaked the behavior of the sendmail method whence it throws the newly added SMTPHeloException (it was closing the connection, which it shouldn't. whatever catches the @@ -34785,7 +34785,7 @@ the "highlights" section above. 4. Changes to the demonstration programs -- Added new useful scipts: byteyears, eptags, fact, from, lfact, +- Added new useful scripts: byteyears, eptags, fact, from, lfact, objgraph, pdeps, pi, primes, ptags, which - Added a bunch of socket demos - Doubled the speed of ptags From webhook-mailer at python.org Thu Oct 28 15:48:43 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:48:43 -0000 Subject: [Python-checkins] bpo-45583: Correct datamodel documentation of int() (GH-29182) Message-ID: https://github.com/python/cpython/commit/d9c1868c25ec6466e8d8ae21fe9315a8a03836ab commit: d9c1868c25ec6466e8d8ae21fe9315a8a03836ab branch: main author: Arthur Milchior committer: ambv date: 2021-10-28T21:48:37+02:00 summary: bpo-45583: Correct datamodel documentation of int() (GH-29182) It should be noted that this part of the documentation is redundant with function.rst's documentation of int. This one was correctly updated with Python 3.8. files: M Doc/reference/datamodel.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index f1334f047d4b7..a6eee22fa332c 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2577,8 +2577,8 @@ left undefined. return the value of the object truncated to an :class:`~numbers.Integral` (typically an :class:`int`). - If :meth:`__int__` is not defined then the built-in function :func:`int` - falls back to :meth:`__trunc__`. + The built-in function :func:`int` falls back to :meth:`__trunc__` if neither + :meth:`__int__` nor :meth:`__index__` is defined. .. _context-managers: From webhook-mailer at python.org Thu Oct 28 15:55:37 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:55:37 -0000 Subject: [Python-checkins] bpo-45612: Add sqlite3 module docstring (GH-29224) Message-ID: https://github.com/python/cpython/commit/4dd1e84789f0bd2da83ad06d23c569bf03713a50 commit: 4dd1e84789f0bd2da83ad06d23c569bf03713a50 branch: main author: Erlend Egeberg Aasland committer: ambv date: 2021-10-28T21:55:27+02:00 summary: bpo-45612: Add sqlite3 module docstring (GH-29224) files: M Lib/sqlite3/__init__.py diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py index f001c0678e195..edc58f15b25ce 100644 --- a/Lib/sqlite3/__init__.py +++ b/Lib/sqlite3/__init__.py @@ -20,6 +20,40 @@ # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. +""" +The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compilant +interface to the SQLite library, and requires SQLite 3.7.15 or newer. + +To use the module, you must first create a database Connection object: + + import sqlite3 + cx = sqlite3.connect("test.db") # test.db will be created or opened + +You can also use the special database name ":memory:" to connect to a transient +in-memory database: + + cx = sqlite3.connect(":memory:") # connect to a database in RAM + +Once you have a Connection object, you can create a Cursor object and call its +execute() method to perform SQL queries: + + cu = cx.cursor() + + # create a table + cu.execute("create table lang(name, first_appeared)") + + # insert values into a table + cu.execute("insert into lang values (?, ?)", ("C", 1972)) + + # execute a query and iterate over the result + for row in cu.execute("select * from lang"): + print(row) + + cx.close() + +The sqlite3 module is written by Gerhard H?ring . +""" + from sqlite3.dbapi2 import * From webhook-mailer at python.org Thu Oct 28 15:55:54 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:55:54 -0000 Subject: [Python-checkins] bpo-45655: Add "relevant PEPs" section to ``typing`` documentation (GH-29280) Message-ID: https://github.com/python/cpython/commit/03db1bbfd2d3f5a343c293b2f0e09a1e962df7ea commit: 03db1bbfd2d3f5a343c293b2f0e09a1e962df7ea branch: main author: Alex Waygood committer: ambv date: 2021-10-28T21:55:50+02:00 summary: bpo-45655: Add "relevant PEPs" section to ``typing`` documentation (GH-29280) The list of PEPs at the top of the documentation for the ``typing`` module has become too long to be readable. This PR proposes presenting this information in a more structured and readable way by adding a new "relevant PEPs" section to the ``typing`` docs. Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index e5e7941833beb..6f501ec136ee6 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -17,13 +17,11 @@ -------------- -This module provides runtime support for type hints as specified by -:pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`, -:pep:`593`, :pep:`612`, :pep:`613` and :pep:`647`. -The most fundamental support consists of the types :data:`Any`, :data:`Union`, -:data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and -:class:`Generic`. For full specification please see :pep:`484`. For -a simplified introduction to type hints see :pep:`483`. +This module provides runtime support for type hints. The most fundamental +support consists of the types :data:`Any`, :data:`Union`, :data:`Tuple`, +:data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For a full +specification, please see :pep:`484`. For a simplified introduction to type +hints, see :pep:`483`. The function below takes and returns a string and is annotated as follows:: @@ -35,6 +33,42 @@ In the function ``greeting``, the argument ``name`` is expected to be of type :class:`str` and the return type :class:`str`. Subtypes are accepted as arguments. +.. _relevant-peps: + +Relevant PEPs +============= + +Since the initial introduction of type hints in :pep:`484` and :pep:`483`, a +number of PEPs have modified and enhanced Python's framework for type +annotations. These include: + +* :pep:`526`: Syntax for Variable Annotations + *Introducing* syntax for annotating variables outside of function + definitions, and :data:`ClassVar` +* :pep:`544`: Protocols: Structural subtyping (static duck typing) + *Introducing* :class:`Protocol` and the + :func:`@runtime_checkable` decorator +* :pep:`585`: Type Hinting Generics In Standard Collections + *Introducing* the ability to use builtin collections and ABCs as + :term:`generic types` +* :pep:`586`: Literal Types + *Introducing* :data:`Literal` +* :pep:`589`: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys + *Introducing* :class:`TypedDict` +* :pep:`591`: Adding a final qualifier to typing + *Introducing* :data:`Final` and the :func:`@final` decorator +* :pep:`593`: Flexible function and variable annotations + *Introducing* :data:`Annotated` +* :pep:`604`: Allow writing union types as ``X | Y`` + *Introducing* :data:`types.UnionType` and the ability to use + the binary-or operator ``|`` as syntactic sugar for a union of types +* :pep:`612`: Parameter Specification Variables + *Introducing* :class:`ParamSpec` and :data:`Concatenate` +* :pep:`613`: Explicit Type Aliases + *Introducing* :data:`TypeAlias` +* :pep:`647`: User-Defined Type Guards + *Introducing* :data:`TypeGuard` + .. _type-aliases: Type aliases diff --git a/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst b/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst new file mode 100644 index 0000000000000..fc5b3d0788817 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst @@ -0,0 +1,2 @@ +Add a new "relevant PEPs" section to the top of the documentation for the +``typing`` module. Patch by Alex Waygood. From webhook-mailer at python.org Thu Oct 28 15:57:19 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:57:19 -0000 Subject: [Python-checkins] bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) (GH-29281) Message-ID: https://github.com/python/cpython/commit/1d88b2b0a15198843a43f34aba1abc65455d3ba7 commit: 1d88b2b0a15198843a43f34aba1abc65455d3ba7 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T21:57:14+02:00 summary: bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) (GH-29281) (cherry picked from commit 88d8a1a340fb09c54d47f354f5fd7d4fbc5f0c78) Co-authored-by: Erlend Egeberg Aasland files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index 5641a347ceda1..dc9aeb2ba3f59 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -117,6 +117,24 @@ Module functions and constants ------------------------------ +.. data:: apilevel + + String constant stating the supported DB-API level. Required by the DB-API. + Hard-coded to ``"2.0"``. + +.. data:: paramstyle + + String constant stating the type of parameter marker formatting expected by + the :mod:`sqlite3` module. Required by the DB-API. Hard-coded to + ``"qmark"``. + + .. note:: + + The :mod:`sqlite3` module supports both ``qmark`` and ``numeric`` DB-API + parameter styles, because that is what the underlying SQLite library + supports. However, the DB-API does not allow multiple values for + the ``paramstyle`` attribute. + .. data:: version The version number of this module, as a string. This is not the version of @@ -139,6 +157,26 @@ Module functions and constants The version number of the run-time SQLite library, as a tuple of integers. +.. data:: threadsafety + + Integer constant required by the DB-API, stating the level of thread safety + the :mod:`sqlite3` module supports. Currently hard-coded to ``1``, meaning + *"Threads may share the module, but not connections."* However, this may not + always be true. You can check the underlying SQLite library's compile-time + threaded mode using the following query:: + + import sqlite3 + con = sqlite3.connect(":memory:") + con.execute(""" + select * from pragma_compile_options + where compile_options like 'THREADSAFE=%' + """).fetchall() + + Note that the `SQLITE_THREADSAFE levels + `_ do not match the DB-API 2.0 + ``threadsafety`` levels. + + .. data:: PARSE_DECLTYPES This constant is meant to be used with the *detect_types* parameter of the @@ -701,6 +739,14 @@ Cursor Objects The cursor will be unusable from this point forward; a :exc:`ProgrammingError` exception will be raised if any operation is attempted with the cursor. + .. method:: setinputsizes(sizes) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + + .. method:: setoutputsize(size [, column]) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + .. attribute:: rowcount Although the :class:`Cursor` class of the :mod:`sqlite3` module implements this From webhook-mailer at python.org Thu Oct 28 15:57:40 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:57:40 -0000 Subject: [Python-checkins] bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) (GH-29282) Message-ID: https://github.com/python/cpython/commit/020aa06ec8b0f473a682f4ae74af5833035b054b commit: 020aa06ec8b0f473a682f4ae74af5833035b054b branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T21:57:35+02:00 summary: bpo-45608: Document missing `sqlite3` DB-API attributes and methods (GH-29219) (GH-29282) (cherry picked from commit 88d8a1a340fb09c54d47f354f5fd7d4fbc5f0c78) Co-authored-by: Erlend Egeberg Aasland files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index aeedcbeffcf4e..b9436daaebb4b 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -116,6 +116,24 @@ Module functions and constants ------------------------------ +.. data:: apilevel + + String constant stating the supported DB-API level. Required by the DB-API. + Hard-coded to ``"2.0"``. + +.. data:: paramstyle + + String constant stating the type of parameter marker formatting expected by + the :mod:`sqlite3` module. Required by the DB-API. Hard-coded to + ``"qmark"``. + + .. note:: + + The :mod:`sqlite3` module supports both ``qmark`` and ``numeric`` DB-API + parameter styles, because that is what the underlying SQLite library + supports. However, the DB-API does not allow multiple values for + the ``paramstyle`` attribute. + .. data:: version The version number of this module, as a string. This is not the version of @@ -138,6 +156,26 @@ Module functions and constants The version number of the run-time SQLite library, as a tuple of integers. +.. data:: threadsafety + + Integer constant required by the DB-API, stating the level of thread safety + the :mod:`sqlite3` module supports. Currently hard-coded to ``1``, meaning + *"Threads may share the module, but not connections."* However, this may not + always be true. You can check the underlying SQLite library's compile-time + threaded mode using the following query:: + + import sqlite3 + con = sqlite3.connect(":memory:") + con.execute(""" + select * from pragma_compile_options + where compile_options like 'THREADSAFE=%' + """).fetchall() + + Note that the `SQLITE_THREADSAFE levels + `_ do not match the DB-API 2.0 + ``threadsafety`` levels. + + .. data:: PARSE_DECLTYPES This constant is meant to be used with the *detect_types* parameter of the @@ -688,6 +726,14 @@ Cursor Objects The cursor will be unusable from this point forward; a :exc:`ProgrammingError` exception will be raised if any operation is attempted with the cursor. + .. method:: setinputsizes(sizes) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + + .. method:: setoutputsize(size [, column]) + + Required by the DB-API. Is a no-op in :mod:`sqlite3`. + .. attribute:: rowcount Although the :class:`Cursor` class of the :mod:`sqlite3` module implements this From webhook-mailer at python.org Thu Oct 28 15:58:28 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 19:58:28 -0000 Subject: [Python-checkins] bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) (GH-29283) Message-ID: https://github.com/python/cpython/commit/fb80aede6ab5d10297b787526657b1a6e20a706a commit: fb80aede6ab5d10297b787526657b1a6e20a706a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T21:58:24+02:00 summary: bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) (GH-29283) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 1fb968c07a76fb2d1ec8c14a0026f1d15828f4a5) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst M Doc/library/multiprocessing.rst diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 7225b65dcb05d..4562128394f88 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2635,12 +2635,13 @@ handler type) for messages from different processes to get mixed up. inherited. .. currentmodule:: multiprocessing -.. function:: log_to_stderr() +.. function:: log_to_stderr(level=None) This function performs a call to :func:`get_logger` but in addition to returning the logger created by get_logger, it adds a handler which sends output to :data:`sys.stderr` using format ``'[%(levelname)s/%(processName)s] %(message)s'``. + You can modify ``levelname`` of the logger by passing a ``level`` argument. Below is an example session with logging turned on:: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst new file mode 100644 index 0000000000000..9da9cca7bf1ea --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst @@ -0,0 +1 @@ +Add ``level`` argument to ``multiprocessing.log_to_stderr`` function docs. \ No newline at end of file From webhook-mailer at python.org Thu Oct 28 16:01:42 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:01:42 -0000 Subject: [Python-checkins] bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) Message-ID: https://github.com/python/cpython/commit/7401694807fc6b5f7b35ff73c06f4bb852e02946 commit: 7401694807fc6b5f7b35ff73c06f4bb852e02946 branch: main author: Carl Friedrich Bolz-Tereick committer: ambv date: 2021-10-28T22:01:35+02:00 summary: bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) the current test depended on integer sets being iterated on in a certain fixed order. That order is different on PyPy (insertion based) and could change in CPython in the future in theory. Make the test robust against a different iteration order by sorting. files: M Lib/test/test_graphlib.py diff --git a/Lib/test/test_graphlib.py b/Lib/test/test_graphlib.py index 00432537f22d0..86246a62ed642 100644 --- a/Lib/test/test_graphlib.py +++ b/Lib/test/test_graphlib.py @@ -13,13 +13,19 @@ def static_order_with_groups(ts): nodes = ts.get_ready() for node in nodes: ts.done(node) - yield nodes + yield tuple(sorted(nodes)) ts = graphlib.TopologicalSorter(graph) self.assertEqual(list(static_order_with_groups(ts)), list(expected)) ts = graphlib.TopologicalSorter(graph) - self.assertEqual(list(ts.static_order()), list(chain(*expected))) + # need to be a bit careful comparing the result of ts.static_order and + # expected, because the order within a group is dependent on set + # iteration order + it = iter(ts.static_order()) + for group in expected: + tsgroup = {next(it) for element in group} + self.assertEqual(set(group), tsgroup) def _assert_cycle(self, graph, cycle): ts = graphlib.TopologicalSorter() @@ -36,7 +42,7 @@ def _assert_cycle(self, graph, cycle): def test_simple_cases(self): self._test_graph( {2: {11}, 9: {11, 8}, 10: {11, 3}, 11: {7, 5}, 8: {7, 3}}, - [(3, 5, 7), (11, 8), (2, 10, 9)], + [(3, 5, 7), (8, 11), (2, 9, 10)], ) self._test_graph({1: {}}, [(1,)]) @@ -80,7 +86,7 @@ def test_no_dependencies(self): def test_the_node_multiple_times(self): # Test same node multiple times in dependencies - self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (1, 3, 0)]) + self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (0, 1, 3)]) # Test adding the same dependency multiple times ts = graphlib.TopologicalSorter() @@ -242,3 +248,6 @@ def check_order_with_hash_seed(seed): self.assertNotEqual(run1, "") self.assertNotEqual(run2, "") self.assertEqual(run1, run2) + +if __name__ == "__main__": + unittest.main() From webhook-mailer at python.org Thu Oct 28 16:02:13 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:02:13 -0000 Subject: [Python-checkins] bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29287) Message-ID: https://github.com/python/cpython/commit/76658e5bdbdf57464f4b43f1625c02c2ba4222dd commit: 76658e5bdbdf57464f4b43f1625c02c2ba4222dd branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:02:07+02:00 summary: bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29287) It should be noted that this part of the documentation is redundant with function.rst's documentation of int. This one was correctly updated with Python 3.8. (cherry picked from commit d9c1868c25ec6466e8d8ae21fe9315a8a03836ab) Co-authored-by: Arthur Milchior files: M Doc/reference/datamodel.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 20d6e326b8b5e..92058aa4ed886 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2507,8 +2507,8 @@ left undefined. return the value of the object truncated to an :class:`~numbers.Integral` (typically an :class:`int`). - If :meth:`__int__` is not defined then the built-in function :func:`int` - falls back to :meth:`__trunc__`. + The built-in function :func:`int` falls back to :meth:`__trunc__` if neither + :meth:`__int__` nor :meth:`__index__` is defined. .. _context-managers: From webhook-mailer at python.org Thu Oct 28 16:03:39 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:03:39 -0000 Subject: [Python-checkins] bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) (GH-29284) Message-ID: https://github.com/python/cpython/commit/01d11b1d62b869f77e024b3979dbc064e9019b7c commit: 01d11b1d62b869f77e024b3979dbc064e9019b7c branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:03:32+02:00 summary: bpo-45604: add `level` argument to `multiprocessing.log_to_stderr` func (GH-29226) (GH-29284) Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot]@users.noreply.github.com> (cherry picked from commit 1fb968c07a76fb2d1ec8c14a0026f1d15828f4a5) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst M Doc/library/multiprocessing.rst diff --git a/Doc/library/multiprocessing.rst b/Doc/library/multiprocessing.rst index 7588e1ddb80a6..2c7d7d8c0e8b1 100644 --- a/Doc/library/multiprocessing.rst +++ b/Doc/library/multiprocessing.rst @@ -2635,12 +2635,13 @@ handler type) for messages from different processes to get mixed up. inherited. .. currentmodule:: multiprocessing -.. function:: log_to_stderr() +.. function:: log_to_stderr(level=None) This function performs a call to :func:`get_logger` but in addition to returning the logger created by get_logger, it adds a handler which sends output to :data:`sys.stderr` using format ``'[%(levelname)s/%(processName)s] %(message)s'``. + You can modify ``levelname`` of the logger by passing a ``level`` argument. Below is an example session with logging turned on:: diff --git a/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst new file mode 100644 index 0000000000000..9da9cca7bf1ea --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-26-10-00-45.bpo-45604.Dm-YhV.rst @@ -0,0 +1 @@ +Add ``level`` argument to ``multiprocessing.log_to_stderr`` function docs. \ No newline at end of file From webhook-mailer at python.org Thu Oct 28 16:17:15 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:17:15 -0000 Subject: [Python-checkins] bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29285) Message-ID: https://github.com/python/cpython/commit/fef54abf5fa3bf3d3cdeb2cba7a2921d204867c6 commit: fef54abf5fa3bf3d3cdeb2cba7a2921d204867c6 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:17:06+02:00 summary: bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29285) It should be noted that this part of the documentation is redundant with function.rst's documentation of int. This one was correctly updated with Python 3.8. (cherry picked from commit d9c1868c25ec6466e8d8ae21fe9315a8a03836ab) Co-authored-by: Arthur Milchior files: M Doc/reference/datamodel.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 310167e86d0cb..195e8c2d16f10 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2541,8 +2541,8 @@ left undefined. return the value of the object truncated to an :class:`~numbers.Integral` (typically an :class:`int`). - If :meth:`__int__` is not defined then the built-in function :func:`int` - falls back to :meth:`__trunc__`. + The built-in function :func:`int` falls back to :meth:`__trunc__` if neither + :meth:`__int__` nor :meth:`__index__` is defined. .. _context-managers: From webhook-mailer at python.org Thu Oct 28 16:17:25 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:17:25 -0000 Subject: [Python-checkins] bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29286) Message-ID: https://github.com/python/cpython/commit/3767e0d94351653a34ba6a6914e57c5c231012b0 commit: 3767e0d94351653a34ba6a6914e57c5c231012b0 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:17:20+02:00 summary: bpo-45583: Correct datamodel documentation of int() (GH-29182) (GH-29286) It should be noted that this part of the documentation is redundant with function.rst's documentation of int. This one was correctly updated with Python 3.8. (cherry picked from commit d9c1868c25ec6466e8d8ae21fe9315a8a03836ab) Co-authored-by: Arthur Milchior files: M Doc/reference/datamodel.rst diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 94f7e5a23b8a4..29a6987054df4 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -2512,8 +2512,8 @@ left undefined. return the value of the object truncated to an :class:`~numbers.Integral` (typically an :class:`int`). - If :meth:`__int__` is not defined then the built-in function :func:`int` - falls back to :meth:`__trunc__`. + The built-in function :func:`int` falls back to :meth:`__trunc__` if neither + :meth:`__int__` nor :meth:`__index__` is defined. .. _context-managers: From webhook-mailer at python.org Thu Oct 28 16:20:15 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:20:15 -0000 Subject: [Python-checkins] bpo-45379: add custom error string for FROZEN_DISABLED (GH-29190) Message-ID: https://github.com/python/cpython/commit/233841ab782953510ad308dc6173072a6cc6a1cd commit: 233841ab782953510ad308dc6173072a6cc6a1cd branch: main author: Filipe La?ns committer: ambv date: 2021-10-28T22:20:07+02:00 summary: bpo-45379: add custom error string for FROZEN_DISABLED (GH-29190) Signed-off-by: Filipe La?ns Co-authored-by: Gareth Rees files: A Misc/NEWS.d/next/Core and Builtins/2021-10-23-13-49-00.bpo-45379.ZF7G3n.rst M Python/import.c diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-10-23-13-49-00.bpo-45379.ZF7G3n.rst b/Misc/NEWS.d/next/Core and Builtins/2021-10-23-13-49-00.bpo-45379.ZF7G3n.rst new file mode 100644 index 0000000000000..f8e2e50dfc3e2 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-10-23-13-49-00.bpo-45379.ZF7G3n.rst @@ -0,0 +1,2 @@ +Clarify :exc:`ImportError` message when we try to explicitly import a +frozen module but frozen modules are disabled. diff --git a/Python/import.c b/Python/import.c index fe4686cd56b3b..15b1956c102df 100644 --- a/Python/import.c +++ b/Python/import.c @@ -1153,9 +1153,11 @@ set_frozen_error(frozen_status status, PyObject *modname) switch (status) { case FROZEN_BAD_NAME: case FROZEN_NOT_FOUND: - case FROZEN_DISABLED: err = "No such frozen object named %R"; break; + case FROZEN_DISABLED: + err = "Frozen modules are disabled and the frozen object named %R is not essential"; + break; case FROZEN_EXCLUDED: err = "Excluded frozen object named %R"; break; From webhook-mailer at python.org Thu Oct 28 16:22:28 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:22:28 -0000 Subject: [Python-checkins] bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) Message-ID: https://github.com/python/cpython/commit/66e6b3dcd3bbab06feeff2cbaf8aade7b6223d6c commit: 66e6b3dcd3bbab06feeff2cbaf8aade7b6223d6c branch: main author: Nikita Sobolev committer: ambv date: 2021-10-28T22:22:24+02:00 summary: bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) files: A Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst M Lib/test/test_zoneinfo/test_zoneinfo.py diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index a9375fd55857b..59b35ef63f987 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -1403,44 +1403,50 @@ def tzpath(self): return [self.zoneinfo_data.tzpath] def test_cache_hit(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) - zi_rt = pickle.loads(pkl) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Is non-pickled ZoneInfo"): - self.assertIs(zi_in, zi_rt) + with self.subTest(test="Is non-pickled ZoneInfo"): + self.assertIs(zi_in, zi_rt) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Is unpickled ZoneInfo"): - self.assertIs(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Is unpickled ZoneInfo"): + self.assertIs(zi_rt, zi_rt2) def test_cache_miss(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) - del zi_in - self.klass.clear_cache() # Induce a cache miss - zi_rt = pickle.loads(pkl) - zi_rt2 = pickle.loads(pkl) + del zi_in + self.klass.clear_cache() # Induce a cache miss + zi_rt = pickle.loads(pkl) + zi_rt2 = pickle.loads(pkl) - self.assertIs(zi_rt, zi_rt2) + self.assertIs(zi_rt, zi_rt2) def test_no_cache(self): - zi_no_cache = self.klass.no_cache("Europe/Dublin") + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_no_cache = self.klass.no_cache("Europe/Dublin") - pkl = pickle.dumps(zi_no_cache) - zi_rt = pickle.loads(pkl) + pkl = pickle.dumps(zi_no_cache, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Not the pickled object"): - self.assertIsNot(zi_rt, zi_no_cache) + with self.subTest(test="Not the pickled object"): + self.assertIsNot(zi_rt, zi_no_cache) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Not a second unpickled object"): - self.assertIsNot(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Not a second unpickled object"): + self.assertIsNot(zi_rt, zi_rt2) - zi_cache = self.klass("Europe/Dublin") - with self.subTest(test="Not a cached object"): - self.assertIsNot(zi_rt, zi_cache) + zi_cache = self.klass("Europe/Dublin") + with self.subTest(test="Not a cached object"): + self.assertIsNot(zi_rt, zi_cache) def test_from_file(self): key = "Europe/Dublin" @@ -1456,35 +1462,38 @@ def test_from_file(self): ] for zi, test_name in test_cases: - with self.subTest(test_name=test_name): - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(test_name=test_name, proto=proto): + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi, protocol=proto) def test_pickle_after_from_file(self): # This may be a bit of paranoia, but this test is to ensure that no # global state is maintained in order to handle the pickle cache and # from_file behavior, and that it is possible to interweave the # constructors of each of these and pickling/unpickling without issues. - key = "Europe/Dublin" - zi = self.klass(key) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + key = "Europe/Dublin" + zi = self.klass(key) - pkl_0 = pickle.dumps(zi) - zi_rt_0 = pickle.loads(pkl_0) - self.assertIs(zi, zi_rt_0) + pkl_0 = pickle.dumps(zi, protocol=proto) + zi_rt_0 = pickle.loads(pkl_0) + self.assertIs(zi, zi_rt_0) - with open(self.zoneinfo_data.path_from_key(key), "rb") as f: - zi_ff = self.klass.from_file(f, key=key) + with open(self.zoneinfo_data.path_from_key(key), "rb") as f: + zi_ff = self.klass.from_file(f, key=key) - pkl_1 = pickle.dumps(zi) - zi_rt_1 = pickle.loads(pkl_1) - self.assertIs(zi, zi_rt_1) + pkl_1 = pickle.dumps(zi, protocol=proto) + zi_rt_1 = pickle.loads(pkl_1) + self.assertIs(zi, zi_rt_1) - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi_ff) + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi_ff, protocol=proto) - pkl_2 = pickle.dumps(zi) - zi_rt_2 = pickle.loads(pkl_2) - self.assertIs(zi, zi_rt_2) + pkl_2 = pickle.dumps(zi, protocol=proto) + zi_rt_2 = pickle.loads(pkl_2) + self.assertIs(zi, zi_rt_2) class CZoneInfoPickleTest(ZoneInfoPickleTest): diff --git a/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst new file mode 100644 index 0000000000000..fc9783eb9ddaa --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst @@ -0,0 +1 @@ +Add subtests for all ``pickle`` protocols in ``test_zoneinfo``. From webhook-mailer at python.org Thu Oct 28 16:25:02 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:25:02 -0000 Subject: [Python-checkins] bpo-45612: Add sqlite3 module docstring (GH-29224) (GH-29288) Message-ID: https://github.com/python/cpython/commit/823b3e39ae12884d5aa3c98341a41b2d6f19d329 commit: 823b3e39ae12884d5aa3c98341a41b2d6f19d329 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:24:52+02:00 summary: bpo-45612: Add sqlite3 module docstring (GH-29224) (GH-29288) (cherry picked from commit 4dd1e84789f0bd2da83ad06d23c569bf03713a50) Co-authored-by: Erlend Egeberg Aasland files: M Lib/sqlite3/__init__.py diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py index f001c0678e195..edc58f15b25ce 100644 --- a/Lib/sqlite3/__init__.py +++ b/Lib/sqlite3/__init__.py @@ -20,6 +20,40 @@ # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. +""" +The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compilant +interface to the SQLite library, and requires SQLite 3.7.15 or newer. + +To use the module, you must first create a database Connection object: + + import sqlite3 + cx = sqlite3.connect("test.db") # test.db will be created or opened + +You can also use the special database name ":memory:" to connect to a transient +in-memory database: + + cx = sqlite3.connect(":memory:") # connect to a database in RAM + +Once you have a Connection object, you can create a Cursor object and call its +execute() method to perform SQL queries: + + cu = cx.cursor() + + # create a table + cu.execute("create table lang(name, first_appeared)") + + # insert values into a table + cu.execute("insert into lang values (?, ?)", ("C", 1972)) + + # execute a query and iterate over the result + for row in cu.execute("select * from lang"): + print(row) + + cx.close() + +The sqlite3 module is written by Gerhard H?ring . +""" + from sqlite3.dbapi2 import * From webhook-mailer at python.org Thu Oct 28 16:25:42 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 20:25:42 -0000 Subject: [Python-checkins] bpo-45612: Add sqlite3 module docstring (GH-29224) (GH-29289) Message-ID: https://github.com/python/cpython/commit/d6623c3ddb9a0e5ffed81253bd40f75c3c662f1a commit: d6623c3ddb9a0e5ffed81253bd40f75c3c662f1a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T22:25:37+02:00 summary: bpo-45612: Add sqlite3 module docstring (GH-29224) (GH-29289) (cherry picked from commit 4dd1e84789f0bd2da83ad06d23c569bf03713a50) Co-authored-by: Erlend Egeberg Aasland files: M Lib/sqlite3/__init__.py diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py index 6c91df27cca70..efdb7f2ba20f1 100644 --- a/Lib/sqlite3/__init__.py +++ b/Lib/sqlite3/__init__.py @@ -20,4 +20,38 @@ # misrepresented as being the original software. # 3. This notice may not be removed or altered from any source distribution. +""" +The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compilant +interface to the SQLite library, and requires SQLite 3.7.15 or newer. + +To use the module, you must first create a database Connection object: + + import sqlite3 + cx = sqlite3.connect("test.db") # test.db will be created or opened + +You can also use the special database name ":memory:" to connect to a transient +in-memory database: + + cx = sqlite3.connect(":memory:") # connect to a database in RAM + +Once you have a Connection object, you can create a Cursor object and call its +execute() method to perform SQL queries: + + cu = cx.cursor() + + # create a table + cu.execute("create table lang(name, first_appeared)") + + # insert values into a table + cu.execute("insert into lang values (?, ?)", ("C", 1972)) + + # execute a query and iterate over the result + for row in cu.execute("select * from lang"): + print(row) + + cx.close() + +The sqlite3 module is written by Gerhard H?ring . +""" + from sqlite3.dbapi2 import * From webhook-mailer at python.org Thu Oct 28 17:04:43 2021 From: webhook-mailer at python.org (ericsnowcurrently) Date: Thu, 28 Oct 2021 21:04:43 -0000 Subject: [Python-checkins] bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778) Message-ID: https://github.com/python/cpython/commit/074fa5750640a067d9894c69378a00ceecc3b948 commit: 074fa5750640a067d9894c69378a00ceecc3b948 branch: main author: Eric Snow committer: ericsnowcurrently date: 2021-10-28T15:04:33-06:00 summary: bpo-45395: Make custom frozen modules additions instead of replacements. (gh-28778) Currently custom modules (the array set on PyImport_FrozenModules) replace all the frozen stdlib modules. That can be problematic and is unlikely to be what the user wants. This change treats the custom frozen modules as additions instead. They take precedence over all other frozen modules except for those needed to bootstrap the import system. If the "code" field of an entry in the custom array is NULL then that frozen module is treated as disabled, which allows a custom entry to disable a frozen stdlib module. This change allows us to get rid of is_essential_frozen_module() and simplifies the logic for which frozen modules should be ignored. https://bugs.python.org/issue45395 files: A Misc/NEWS.d/next/C API/2021-10-06-15-54-40.bpo-45395.yVhdAl.rst M Doc/library/ctypes.rst M Include/internal/pycore_import.h M Lib/ctypes/test/test_values.py M Programs/_freeze_module.c M Programs/_testembed.c M Python/frozen.c M Python/import.c M Tools/freeze/freeze.py M Tools/scripts/freeze_modules.py diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 87c9c66edd67c..74611144123b0 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -1095,7 +1095,7 @@ We have defined the :c:type:`struct _frozen` data type, so we can get the pointe to the table:: >>> FrozenTable = POINTER(struct_frozen) - >>> table = FrozenTable.in_dll(pythonapi, "PyImport_FrozenModules") + >>> table = FrozenTable.in_dll(pythonapi, "_PyImport_FrozenBootstrap") >>> Since ``table`` is a ``pointer`` to the array of ``struct_frozen`` records, we @@ -1111,9 +1111,7 @@ hit the ``NULL`` entry:: ... _frozen_importlib 31764 _frozen_importlib_external 41499 - __hello__ 161 - __phello__ -161 - __phello__.spam 161 + zipimport 12345 >>> The fact that standard Python has a frozen module and a frozen package diff --git a/Include/internal/pycore_import.h b/Include/internal/pycore_import.h index 6439b7369fb59..aee1f66a3ea17 100644 --- a/Include/internal/pycore_import.h +++ b/Include/internal/pycore_import.h @@ -15,6 +15,9 @@ struct _module_alias { const char *orig; /* ASCII encoded string */ }; +PyAPI_DATA(const struct _frozen *) _PyImport_FrozenBootstrap; +PyAPI_DATA(const struct _frozen *) _PyImport_FrozenStdlib; +PyAPI_DATA(const struct _frozen *) _PyImport_FrozenTest; extern const struct _module_alias * _PyImport_FrozenAliases; #ifdef __cplusplus diff --git a/Lib/ctypes/test/test_values.py b/Lib/ctypes/test/test_values.py index 96521fd3abce9..5f9fa066c4a41 100644 --- a/Lib/ctypes/test/test_values.py +++ b/Lib/ctypes/test/test_values.py @@ -56,35 +56,37 @@ class struct_frozen(Structure): ("size", c_int)] FrozenTable = POINTER(struct_frozen) - ft = FrozenTable.in_dll(pythonapi, "PyImport_FrozenModules") - # ft is a pointer to the struct_frozen entries: modules = [] - for entry in ft: - # This is dangerous. We *can* iterate over a pointer, but - # the loop will not terminate (maybe with an access - # violation;-) because the pointer instance has no size. - if entry.name is None: - break - modname = entry.name.decode("ascii") - modules.append(modname) - with self.subTest(modname): - # Do a sanity check on entry.size and entry.code. - self.assertGreater(abs(entry.size), 10) - self.assertTrue([entry.code[i] for i in range(abs(entry.size))]) - # Check the module's package-ness. - with import_helper.frozen_modules(): - spec = importlib.util.find_spec(modname) - if entry.size < 0: - # It's a package. - self.assertIsNotNone(spec.submodule_search_locations) - else: - self.assertIsNone(spec.submodule_search_locations) + for group in ["Bootstrap", "Stdlib", "Test"]: + ft = FrozenTable.in_dll(pythonapi, f"_PyImport_Frozen{group}") + # ft is a pointer to the struct_frozen entries: + for entry in ft: + # This is dangerous. We *can* iterate over a pointer, but + # the loop will not terminate (maybe with an access + # violation;-) because the pointer instance has no size. + if entry.name is None: + break + modname = entry.name.decode("ascii") + modules.append(modname) + with self.subTest(modname): + # Do a sanity check on entry.size and entry.code. + self.assertGreater(abs(entry.size), 10) + self.assertTrue([entry.code[i] for i in range(abs(entry.size))]) + # Check the module's package-ness. + with import_helper.frozen_modules(): + spec = importlib.util.find_spec(modname) + if entry.size < 0: + # It's a package. + self.assertIsNotNone(spec.submodule_search_locations) + else: + self.assertIsNone(spec.submodule_search_locations) with import_helper.frozen_modules(): expected = _imp._frozen_module_names() self.maxDiff = None - self.assertEqual(modules, expected, "PyImport_FrozenModules example " - "in Doc/library/ctypes.rst may be out of date") + self.assertEqual(modules, expected, + "_PyImport_FrozenBootstrap example " + "in Doc/library/ctypes.rst may be out of date") from ctypes import _pointer_type_cache del _pointer_type_cache[struct_frozen] diff --git a/Misc/NEWS.d/next/C API/2021-10-06-15-54-40.bpo-45395.yVhdAl.rst b/Misc/NEWS.d/next/C API/2021-10-06-15-54-40.bpo-45395.yVhdAl.rst new file mode 100644 index 0000000000000..8996513dc9fb0 --- /dev/null +++ b/Misc/NEWS.d/next/C API/2021-10-06-15-54-40.bpo-45395.yVhdAl.rst @@ -0,0 +1,4 @@ +Custom frozen modules (the array set to ``PyImport_FrozenModules``) are now +treated as additions, rather than replacing all the default frozen modules. +Frozen stdlib modules can still be disabled by setting the "code" field of +the custom array entry to NULL. diff --git a/Programs/_freeze_module.c b/Programs/_freeze_module.c index 316c70d2c7824..e3f6c11c8b073 100644 --- a/Programs/_freeze_module.c +++ b/Programs/_freeze_module.c @@ -23,13 +23,16 @@ of frozen modules instead, left deliberately blank so as to avoid unintentional import of a stale version of _frozen_importlib. */ -static const struct _frozen _PyImport_FrozenModules[] = { +static const struct _frozen no_modules[] = { {0, 0, 0} /* sentinel */ }; static const struct _module_alias aliases[] = { {0, 0} /* sentinel */ }; +const struct _frozen *_PyImport_FrozenBootstrap; +const struct _frozen *_PyImport_FrozenStdlib; +const struct _frozen *_PyImport_FrozenTest; const struct _frozen *PyImport_FrozenModules; const struct _module_alias *_PyImport_FrozenAliases; @@ -188,7 +191,10 @@ main(int argc, char *argv[]) { const char *name, *inpath, *outpath; - PyImport_FrozenModules = _PyImport_FrozenModules; + _PyImport_FrozenBootstrap = no_modules; + _PyImport_FrozenStdlib = no_modules; + _PyImport_FrozenTest = no_modules; + PyImport_FrozenModules = NULL; _PyImport_FrozenAliases = aliases; if (argc != 4) { diff --git a/Programs/_testembed.c b/Programs/_testembed.c index 773c6c3e9900a..6fe18d93a73ae 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -8,6 +8,7 @@ #include #include "pycore_initconfig.h" // _PyConfig_InitCompatConfig() #include "pycore_runtime.h" // _PyRuntime +#include "pycore_import.h" // _PyImport_FrozenBootstrap #include #include #include @@ -1804,30 +1805,10 @@ static int test_unicode_id_init(void) static int test_frozenmain(void) { - // Get "_frozen_importlib" and "_frozen_importlib_external" - // from PyImport_FrozenModules - const struct _frozen *importlib = NULL, *importlib_external = NULL; - for (const struct _frozen *mod = PyImport_FrozenModules; mod->name != NULL; mod++) { - if (strcmp(mod->name, "_frozen_importlib") == 0) { - importlib = mod; - } - else if (strcmp(mod->name, "_frozen_importlib_external") == 0) { - importlib_external = mod; - } - } - if (importlib == NULL || importlib_external == NULL) { - error("cannot find frozen importlib and importlib_external"); - return 1; - } - static struct _frozen frozen_modules[4] = { - {0, 0, 0}, // importlib - {0, 0, 0}, // importlib_external {"__main__", M_test_frozenmain, sizeof(M_test_frozenmain)}, {0, 0, 0} // sentinel }; - frozen_modules[0] = *importlib; - frozen_modules[1] = *importlib_external; char* argv[] = { "./argv0", @@ -1846,7 +1827,12 @@ static int test_frozenmain(void) static int list_frozen(void) { const struct _frozen *p; - for (p = PyImport_FrozenModules; ; p++) { + for (p = _PyImport_FrozenBootstrap; ; p++) { + if (p->name == NULL) + break; + printf("%s\n", p->name); + } + for (p = _PyImport_FrozenStdlib; ; p++) { if (p->name == NULL) break; printf("%s\n", p->name); diff --git a/Python/frozen.c b/Python/frozen.c index 499b3b9957057..15baa97b9d055 100644 --- a/Python/frozen.c +++ b/Python/frozen.c @@ -63,14 +63,15 @@ /* Note that a negative size indicates a package. */ -static const struct _frozen _PyImport_FrozenModules[] = { - /* import system */ +static const struct _frozen bootstrap_modules[] = { {"_frozen_importlib", _Py_M__importlib__bootstrap, (int)sizeof(_Py_M__importlib__bootstrap)}, {"_frozen_importlib_external", _Py_M__importlib__bootstrap_external, (int)sizeof(_Py_M__importlib__bootstrap_external)}, {"zipimport", _Py_M__zipimport, (int)sizeof(_Py_M__zipimport)}, - + {0, 0, 0} /* bootstrap sentinel */ +}; +static const struct _frozen stdlib_modules[] = { /* stdlib - startup, without site (python -S) */ {"abc", _Py_M__abc, (int)sizeof(_Py_M__abc)}, {"codecs", _Py_M__codecs, (int)sizeof(_Py_M__codecs)}, @@ -87,8 +88,9 @@ static const struct _frozen _PyImport_FrozenModules[] = { {"os", _Py_M__os, (int)sizeof(_Py_M__os)}, {"site", _Py_M__site, (int)sizeof(_Py_M__site)}, {"stat", _Py_M__stat, (int)sizeof(_Py_M__stat)}, - - /* Test module */ + {0, 0, 0} /* stdlib sentinel */ +}; +static const struct _frozen test_modules[] = { {"__hello__", _Py_M____hello__, (int)sizeof(_Py_M____hello__)}, {"__hello_alias__", _Py_M____hello__, (int)sizeof(_Py_M____hello__)}, {"__phello_alias__", _Py_M____hello__, -(int)sizeof(_Py_M____hello__)}, @@ -103,8 +105,11 @@ static const struct _frozen _PyImport_FrozenModules[] = { {"__phello__.spam", _Py_M____phello___spam, (int)sizeof(_Py_M____phello___spam)}, {"__hello_only__", _Py_M__frozen_only, (int)sizeof(_Py_M__frozen_only)}, - {0, 0, 0} /* modules sentinel */ + {0, 0, 0} /* test sentinel */ }; +const struct _frozen *_PyImport_FrozenBootstrap = bootstrap_modules; +const struct _frozen *_PyImport_FrozenStdlib = stdlib_modules; +const struct _frozen *_PyImport_FrozenTest = test_modules; static const struct _module_alias aliases[] = { {"_frozen_importlib", "importlib._bootstrap"}, @@ -124,4 +129,4 @@ const struct _module_alias *_PyImport_FrozenAliases = aliases; /* Embedding apps may change this pointer to point to their favorite collection of frozen modules: */ -const struct _frozen *PyImport_FrozenModules = _PyImport_FrozenModules; +const struct _frozen *PyImport_FrozenModules = NULL; diff --git a/Python/import.c b/Python/import.c index 15b1956c102df..48ea9129163c8 100644 --- a/Python/import.c +++ b/Python/import.c @@ -1070,27 +1070,6 @@ resolve_module_alias(const char *name, const struct _module_alias *aliases, /* Frozen modules */ -static bool -is_essential_frozen_module(const char *name) -{ - /* These modules are necessary to bootstrap the import system. */ - if (strcmp(name, "_frozen_importlib") == 0) { - return true; - } - if (strcmp(name, "_frozen_importlib_external") == 0) { - return true; - } - if (strcmp(name, "zipimport") == 0) { - return true; - } - /* This doesn't otherwise have anywhere to find the module. - See frozenmain.c. */ - if (strcmp(name, "__main__") == 0) { - return true; - } - return false; -} - static bool use_frozen(void) { @@ -1115,26 +1094,76 @@ list_frozen_module_names() return NULL; } bool enabled = use_frozen(); - for (const struct _frozen *p = PyImport_FrozenModules; ; p++) { + const struct _frozen *p; +#define ADD_MODULE(name) \ + do { \ + PyObject *nameobj = PyUnicode_FromString(name); \ + if (nameobj == NULL) { \ + goto error; \ + } \ + int res = PyList_Append(names, nameobj); \ + Py_DECREF(nameobj); \ + if (res != 0) { \ + goto error; \ + } \ + } while(0) + // We always use the bootstrap modules. + for (p = _PyImport_FrozenBootstrap; ; p++) { if (p->name == NULL) { break; } - if (!enabled && !is_essential_frozen_module(p->name)) { - continue; + ADD_MODULE(p->name); + } + // Frozen stdlib modules may be disabled. + for (p = _PyImport_FrozenStdlib; ; p++) { + if (p->name == NULL) { + break; } - PyObject *name = PyUnicode_FromString(p->name); - if (name == NULL) { - Py_DECREF(names); - return NULL; + if (enabled) { + ADD_MODULE(p->name); } - int res = PyList_Append(names, name); - Py_DECREF(name); - if (res != 0) { - Py_DECREF(names); - return NULL; + } + for (p = _PyImport_FrozenTest; ; p++) { + if (p->name == NULL) { + break; + } + if (enabled) { + ADD_MODULE(p->name); + } + } +#undef ADD_MODULE + // Add any custom modules. + if (PyImport_FrozenModules != NULL) { + for (p = PyImport_FrozenModules; ; p++) { + if (p->name == NULL) { + break; + } + PyObject *nameobj = PyUnicode_FromString(p->name); + if (nameobj == NULL) { + goto error; + } + int found = PySequence_Contains(names, nameobj); + if (found < 0) { + Py_DECREF(nameobj); + goto error; + } + else if (found) { + Py_DECREF(nameobj); + } + else { + int res = PyList_Append(names, nameobj); + Py_DECREF(nameobj); + if (res != 0) { + goto error; + } + } } } return names; + +error: + Py_DECREF(names); + return NULL; } typedef enum { @@ -1180,6 +1209,54 @@ set_frozen_error(frozen_status status, PyObject *modname) } } +static const struct _frozen * +look_up_frozen(const char *name) +{ + const struct _frozen *p; + // We always use the bootstrap modules. + for (p = _PyImport_FrozenBootstrap; ; p++) { + if (p->name == NULL) { + // We hit the end-of-list sentinel value. + break; + } + if (strcmp(name, p->name) == 0) { + return p; + } + } + // Prefer custom modules, if any. Frozen stdlib modules can be + // disabled here by setting "code" to NULL in the array entry. + if (PyImport_FrozenModules != NULL) { + for (p = PyImport_FrozenModules; ; p++) { + if (p->name == NULL) { + break; + } + if (strcmp(name, p->name) == 0) { + return p; + } + } + } + // Frozen stdlib modules may be disabled. + if (use_frozen()) { + for (p = _PyImport_FrozenStdlib; ; p++) { + if (p->name == NULL) { + break; + } + if (strcmp(name, p->name) == 0) { + return p; + } + } + for (p = _PyImport_FrozenTest; ; p++) { + if (p->name == NULL) { + break; + } + if (strcmp(name, p->name) == 0) { + return p; + } + } + } + return NULL; +} + struct frozen_info { PyObject *nameobj; const char *data; @@ -1209,19 +1286,9 @@ find_frozen(PyObject *nameobj, struct frozen_info *info) return FROZEN_BAD_NAME; } - if (!use_frozen() && !is_essential_frozen_module(name)) { - return FROZEN_DISABLED; - } - - const struct _frozen *p; - for (p = PyImport_FrozenModules; ; p++) { - if (p->name == NULL) { - // We hit the end-of-list sentinel value. - return FROZEN_NOT_FOUND; - } - if (strcmp(name, p->name) == 0) { - break; - } + const struct _frozen *p = look_up_frozen(name); + if (p == NULL) { + return FROZEN_NOT_FOUND; } if (info != NULL) { info->nameobj = nameobj; // borrowed diff --git a/Tools/freeze/freeze.py b/Tools/freeze/freeze.py index d66e1e2708e75..bc5e43f4853de 100755 --- a/Tools/freeze/freeze.py +++ b/Tools/freeze/freeze.py @@ -367,12 +367,6 @@ def main(): else: mf.load_file(mod) - # Alias "importlib._bootstrap" to "_frozen_importlib" so that the - # import machinery can bootstrap. Do the same for - # importlib._bootstrap_external. - mf.modules["_frozen_importlib"] = mf.modules["importlib._bootstrap"] - mf.modules["_frozen_importlib_external"] = mf.modules["importlib._bootstrap_external"] - # Add the main script as either __main__, or the actual module name. if python_entry_is_main: mf.run_script(scriptfile) diff --git a/Tools/scripts/freeze_modules.py b/Tools/scripts/freeze_modules.py index 5c7eee4295289..36142625ca609 100644 --- a/Tools/scripts/freeze_modules.py +++ b/Tools/scripts/freeze_modules.py @@ -60,6 +60,7 @@ def find_tool(): OS_PATH = 'ntpath' if os.name == 'nt' else 'posixpath' # These are modules that get frozen. +TESTS_SECTION = 'Test module' FROZEN = [ # See parse_frozen_spec() for the format. # In cases where the frozenid is duplicated, the first one is re-used. @@ -94,7 +95,7 @@ def find_tool(): 'site', 'stat', ]), - ('Test module', [ + (TESTS_SECTION, [ '__hello__', '__hello__ : __hello_alias__', '__hello__ : <__phello_alias__>', @@ -103,7 +104,7 @@ def find_tool(): f'frozen_only : __hello_only__ = {FROZEN_ONLY}', ]), ] -ESSENTIAL = { +BOOTSTRAP = { 'importlib._bootstrap', 'importlib._bootstrap_external', 'zipimport', @@ -527,16 +528,24 @@ def regen_frozen(modules): header = relpath_for_posix_display(src.frozenfile, parentdir) headerlines.append(f'#include "{header}"') - deflines = [] + bootstraplines = [] + stdliblines = [] + testlines = [] aliaslines = [] indent = ' ' lastsection = None for mod in modules: - if mod.section != lastsection: - if lastsection is not None: - deflines.append('') - deflines.append(f'/* {mod.section} */') - lastsection = mod.section + if mod.frozenid in BOOTSTRAP: + lines = bootstraplines + elif mod.section == TESTS_SECTION: + lines = testlines + else: + lines = stdliblines + if mod.section != lastsection: + if lastsection is not None: + lines.append('') + lines.append(f'/* {mod.section} */') + lastsection = mod.section symbol = mod.symbol pkg = '-' if mod.ispkg else '' @@ -544,11 +553,11 @@ def regen_frozen(modules): ) % (mod.name, symbol, pkg, symbol) # TODO: Consider not folding lines if len(line) < 80: - deflines.append(line) + lines.append(line) else: line1, _, line2 = line.rpartition(' ') - deflines.append(line1) - deflines.append(indent + line2) + lines.append(line1) + lines.append(indent + line2) if mod.isalias: if not mod.orig: @@ -559,11 +568,13 @@ def regen_frozen(modules): entry = '{"%s", "%s"},' % (mod.name, mod.orig) aliaslines.append(indent + entry) - if not deflines[0]: - del deflines[0] - for i, line in enumerate(deflines): - if line: - deflines[i] = indent + line + for lines in (bootstraplines, stdliblines, testlines): + # TODO: Is this necessary any more? + if not lines[0]: + del lines[0] + for i, line in enumerate(lines): + if line: + lines[i] = indent + line print(f'# Updating {os.path.relpath(FROZEN_FILE)}') with updating_file_with_tmpfile(FROZEN_FILE) as (infile, outfile): @@ -579,9 +590,23 @@ def regen_frozen(modules): ) lines = replace_block( lines, - "static const struct _frozen _PyImport_FrozenModules[] =", - "/* modules sentinel */", - deflines, + "static const struct _frozen bootstrap_modules[] =", + "/* bootstrap sentinel */", + bootstraplines, + FROZEN_FILE, + ) + lines = replace_block( + lines, + "static const struct _frozen stdlib_modules[] =", + "/* stdlib sentinel */", + stdliblines, + FROZEN_FILE, + ) + lines = replace_block( + lines, + "static const struct _frozen test_modules[] =", + "/* test sentinel */", + testlines, FROZEN_FILE, ) lines = replace_block( From webhook-mailer at python.org Thu Oct 28 17:14:45 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 21:14:45 -0000 Subject: [Python-checkins] bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) (GH-29292) Message-ID: https://github.com/python/cpython/commit/67a1abb6aab3b3ce40eb3fdc0af73179ab436a3a commit: 67a1abb6aab3b3ce40eb3fdc0af73179ab436a3a branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T23:14:37+02:00 summary: bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) (GH-29292) the current test depended on integer sets being iterated on in a certain fixed order. That order is different on PyPy (insertion based) and could change in CPython in the future in theory. Make the test robust against a different iteration order by sorting. (cherry picked from commit 7401694807fc6b5f7b35ff73c06f4bb852e02946) Co-authored-by: Carl Friedrich Bolz-Tereick files: M Lib/test/test_graphlib.py diff --git a/Lib/test/test_graphlib.py b/Lib/test/test_graphlib.py index 00432537f22d0..86246a62ed642 100644 --- a/Lib/test/test_graphlib.py +++ b/Lib/test/test_graphlib.py @@ -13,13 +13,19 @@ def static_order_with_groups(ts): nodes = ts.get_ready() for node in nodes: ts.done(node) - yield nodes + yield tuple(sorted(nodes)) ts = graphlib.TopologicalSorter(graph) self.assertEqual(list(static_order_with_groups(ts)), list(expected)) ts = graphlib.TopologicalSorter(graph) - self.assertEqual(list(ts.static_order()), list(chain(*expected))) + # need to be a bit careful comparing the result of ts.static_order and + # expected, because the order within a group is dependent on set + # iteration order + it = iter(ts.static_order()) + for group in expected: + tsgroup = {next(it) for element in group} + self.assertEqual(set(group), tsgroup) def _assert_cycle(self, graph, cycle): ts = graphlib.TopologicalSorter() @@ -36,7 +42,7 @@ def _assert_cycle(self, graph, cycle): def test_simple_cases(self): self._test_graph( {2: {11}, 9: {11, 8}, 10: {11, 3}, 11: {7, 5}, 8: {7, 3}}, - [(3, 5, 7), (11, 8), (2, 10, 9)], + [(3, 5, 7), (8, 11), (2, 9, 10)], ) self._test_graph({1: {}}, [(1,)]) @@ -80,7 +86,7 @@ def test_no_dependencies(self): def test_the_node_multiple_times(self): # Test same node multiple times in dependencies - self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (1, 3, 0)]) + self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (0, 1, 3)]) # Test adding the same dependency multiple times ts = graphlib.TopologicalSorter() @@ -242,3 +248,6 @@ def check_order_with_hash_seed(seed): self.assertNotEqual(run1, "") self.assertNotEqual(run2, "") self.assertEqual(run1, run2) + +if __name__ == "__main__": + unittest.main() From webhook-mailer at python.org Thu Oct 28 17:15:06 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 21:15:06 -0000 Subject: [Python-checkins] bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) (GH-29293) Message-ID: https://github.com/python/cpython/commit/eccb753ae6e1459dc697d5408e1082fff4f6d8f7 commit: eccb753ae6e1459dc697d5408e1082fff4f6d8f7 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-28T23:15:01+02:00 summary: bpo-45624: make test_graphlib not depend on the iteration order of sets (GH-29233) (GH-29293) the current test depended on integer sets being iterated on in a certain fixed order. That order is different on PyPy (insertion based) and could change in CPython in the future in theory. Make the test robust against a different iteration order by sorting. (cherry picked from commit 7401694807fc6b5f7b35ff73c06f4bb852e02946) Co-authored-by: Carl Friedrich Bolz-Tereick files: M Lib/test/test_graphlib.py diff --git a/Lib/test/test_graphlib.py b/Lib/test/test_graphlib.py index 00432537f22d0..86246a62ed642 100644 --- a/Lib/test/test_graphlib.py +++ b/Lib/test/test_graphlib.py @@ -13,13 +13,19 @@ def static_order_with_groups(ts): nodes = ts.get_ready() for node in nodes: ts.done(node) - yield nodes + yield tuple(sorted(nodes)) ts = graphlib.TopologicalSorter(graph) self.assertEqual(list(static_order_with_groups(ts)), list(expected)) ts = graphlib.TopologicalSorter(graph) - self.assertEqual(list(ts.static_order()), list(chain(*expected))) + # need to be a bit careful comparing the result of ts.static_order and + # expected, because the order within a group is dependent on set + # iteration order + it = iter(ts.static_order()) + for group in expected: + tsgroup = {next(it) for element in group} + self.assertEqual(set(group), tsgroup) def _assert_cycle(self, graph, cycle): ts = graphlib.TopologicalSorter() @@ -36,7 +42,7 @@ def _assert_cycle(self, graph, cycle): def test_simple_cases(self): self._test_graph( {2: {11}, 9: {11, 8}, 10: {11, 3}, 11: {7, 5}, 8: {7, 3}}, - [(3, 5, 7), (11, 8), (2, 10, 9)], + [(3, 5, 7), (8, 11), (2, 9, 10)], ) self._test_graph({1: {}}, [(1,)]) @@ -80,7 +86,7 @@ def test_no_dependencies(self): def test_the_node_multiple_times(self): # Test same node multiple times in dependencies - self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (1, 3, 0)]) + self._test_graph({1: {2}, 3: {4}, 0: [2, 4, 4, 4, 4, 4]}, [(2, 4), (0, 1, 3)]) # Test adding the same dependency multiple times ts = graphlib.TopologicalSorter() @@ -242,3 +248,6 @@ def check_order_with_hash_seed(seed): self.assertNotEqual(run1, "") self.assertNotEqual(run2, "") self.assertEqual(run1, run2) + +if __name__ == "__main__": + unittest.main() From webhook-mailer at python.org Thu Oct 28 17:17:33 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 21:17:33 -0000 Subject: [Python-checkins] [3.9] bpo-45655: Add "relevant PEPs" section to typing documentation (GH-29297) Message-ID: https://github.com/python/cpython/commit/8813a987b1df78b5eaddb085e514dfa5af5c8634 commit: 8813a987b1df78b5eaddb085e514dfa5af5c8634 branch: 3.9 author: Alex Waygood committer: ambv date: 2021-10-28T23:17:28+02:00 summary: [3.9] bpo-45655: Add "relevant PEPs" section to typing documentation (GH-29297) Backport of https://github.com/python/cpython/pull/29280 to the 3.9 branch. files: A Misc/NEWS.d/next/Documentation/2021-10-28-21-28-07.bpo-45655.aPYGaS.rst M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 78ab760d86a54..295f2ae8905af 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -17,12 +17,11 @@ -------------- -This module provides runtime support for type hints as specified by -:pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, and :pep:`591`. -The most fundamental support consists of the types :data:`Any`, :data:`Union`, -:data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and -:class:`Generic`. For full specification please see :pep:`484`. For -a simplified introduction to type hints see :pep:`483`. +This module provides runtime support for type hints. The most fundamental +support consists of the types :data:`Any`, :data:`Union`, :data:`Tuple`, +:data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For a full +specification, please see :pep:`484`. For a simplified introduction to type +hints, see :pep:`483`. The function below takes and returns a string and is annotated as follows:: @@ -34,6 +33,33 @@ In the function ``greeting``, the argument ``name`` is expected to be of type :class:`str` and the return type :class:`str`. Subtypes are accepted as arguments. +.. _relevant-peps: + +Relevant PEPs +============= + +Since the initial introduction of type hints in :pep:`484` and :pep:`483`, a +number of PEPs have modified and enhanced Python's framework for type +annotations. These include: + +* :pep:`526`: Syntax for Variable Annotations + *Introducing* syntax for annotating variables outside of function + definitions, and :data:`ClassVar` +* :pep:`544`: Protocols: Structural subtyping (static duck typing) + *Introducing* :class:`Protocol` and the + :func:`@runtime_checkable` decorator +* :pep:`585`: Type Hinting Generics In Standard Collections + *Introducing* the ability to use builtin collections and ABCs as + :term:`generic types` +* :pep:`586`: Literal Types + *Introducing* :class:`Literal` +* :pep:`589`: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys + *Introducing* :class:`TypedDict` +* :pep:`591`: Adding a final qualifier to typing + *Introducing* :data:`Final` and the :func:`@final` decorator +* :pep:`593`: Flexible function and variable annotations + *Introducing* :class:`Annotated` + Type aliases ============ diff --git a/Misc/NEWS.d/next/Documentation/2021-10-28-21-28-07.bpo-45655.aPYGaS.rst b/Misc/NEWS.d/next/Documentation/2021-10-28-21-28-07.bpo-45655.aPYGaS.rst new file mode 100644 index 0000000000000..fc5b3d0788817 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-28-21-28-07.bpo-45655.aPYGaS.rst @@ -0,0 +1,2 @@ +Add a new "relevant PEPs" section to the top of the documentation for the +``typing`` module. Patch by Alex Waygood. From webhook-mailer at python.org Thu Oct 28 18:00:42 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 22:00:42 -0000 Subject: [Python-checkins] bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) (GH-29295) Message-ID: https://github.com/python/cpython/commit/dd674ca96f2150fb3f7b4086ef7ec0022c4e2058 commit: dd674ca96f2150fb3f7b4086ef7ec0022c4e2058 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T00:00:34+02:00 summary: bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) (GH-29295) (cherry picked from commit 66e6b3dcd3bbab06feeff2cbaf8aade7b6223d6c) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst M Lib/test/test_zoneinfo/test_zoneinfo.py diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index d4704b75d389d..08ebbb1dd9911 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -1403,44 +1403,50 @@ def tzpath(self): return [self.zoneinfo_data.tzpath] def test_cache_hit(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) - zi_rt = pickle.loads(pkl) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Is non-pickled ZoneInfo"): - self.assertIs(zi_in, zi_rt) + with self.subTest(test="Is non-pickled ZoneInfo"): + self.assertIs(zi_in, zi_rt) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Is unpickled ZoneInfo"): - self.assertIs(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Is unpickled ZoneInfo"): + self.assertIs(zi_rt, zi_rt2) def test_cache_miss(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) - del zi_in - self.klass.clear_cache() # Induce a cache miss - zi_rt = pickle.loads(pkl) - zi_rt2 = pickle.loads(pkl) + del zi_in + self.klass.clear_cache() # Induce a cache miss + zi_rt = pickle.loads(pkl) + zi_rt2 = pickle.loads(pkl) - self.assertIs(zi_rt, zi_rt2) + self.assertIs(zi_rt, zi_rt2) def test_no_cache(self): - zi_no_cache = self.klass.no_cache("Europe/Dublin") + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_no_cache = self.klass.no_cache("Europe/Dublin") - pkl = pickle.dumps(zi_no_cache) - zi_rt = pickle.loads(pkl) + pkl = pickle.dumps(zi_no_cache, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Not the pickled object"): - self.assertIsNot(zi_rt, zi_no_cache) + with self.subTest(test="Not the pickled object"): + self.assertIsNot(zi_rt, zi_no_cache) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Not a second unpickled object"): - self.assertIsNot(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Not a second unpickled object"): + self.assertIsNot(zi_rt, zi_rt2) - zi_cache = self.klass("Europe/Dublin") - with self.subTest(test="Not a cached object"): - self.assertIsNot(zi_rt, zi_cache) + zi_cache = self.klass("Europe/Dublin") + with self.subTest(test="Not a cached object"): + self.assertIsNot(zi_rt, zi_cache) def test_from_file(self): key = "Europe/Dublin" @@ -1456,35 +1462,38 @@ def test_from_file(self): ] for zi, test_name in test_cases: - with self.subTest(test_name=test_name): - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(test_name=test_name, proto=proto): + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi, protocol=proto) def test_pickle_after_from_file(self): # This may be a bit of paranoia, but this test is to ensure that no # global state is maintained in order to handle the pickle cache and # from_file behavior, and that it is possible to interweave the # constructors of each of these and pickling/unpickling without issues. - key = "Europe/Dublin" - zi = self.klass(key) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + key = "Europe/Dublin" + zi = self.klass(key) - pkl_0 = pickle.dumps(zi) - zi_rt_0 = pickle.loads(pkl_0) - self.assertIs(zi, zi_rt_0) + pkl_0 = pickle.dumps(zi, protocol=proto) + zi_rt_0 = pickle.loads(pkl_0) + self.assertIs(zi, zi_rt_0) - with open(self.zoneinfo_data.path_from_key(key), "rb") as f: - zi_ff = self.klass.from_file(f, key=key) + with open(self.zoneinfo_data.path_from_key(key), "rb") as f: + zi_ff = self.klass.from_file(f, key=key) - pkl_1 = pickle.dumps(zi) - zi_rt_1 = pickle.loads(pkl_1) - self.assertIs(zi, zi_rt_1) + pkl_1 = pickle.dumps(zi, protocol=proto) + zi_rt_1 = pickle.loads(pkl_1) + self.assertIs(zi, zi_rt_1) - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi_ff) + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi_ff, protocol=proto) - pkl_2 = pickle.dumps(zi) - zi_rt_2 = pickle.loads(pkl_2) - self.assertIs(zi, zi_rt_2) + pkl_2 = pickle.dumps(zi, protocol=proto) + zi_rt_2 = pickle.loads(pkl_2) + self.assertIs(zi, zi_rt_2) class CZoneInfoPickleTest(ZoneInfoPickleTest): diff --git a/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst new file mode 100644 index 0000000000000..fc9783eb9ddaa --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst @@ -0,0 +1 @@ +Add subtests for all ``pickle`` protocols in ``test_zoneinfo``. From webhook-mailer at python.org Thu Oct 28 18:00:54 2021 From: webhook-mailer at python.org (ambv) Date: Thu, 28 Oct 2021 22:00:54 -0000 Subject: [Python-checkins] bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) (GH-29296) Message-ID: https://github.com/python/cpython/commit/7203ecd332eca3a44a3f1c8bdadd76a08c5568a1 commit: 7203ecd332eca3a44a3f1c8bdadd76a08c5568a1 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T00:00:49+02:00 summary: bpo-45577: test all pickle protocols in `test_zoneinfo` (GH-29167) (GH-29296) (cherry picked from commit 66e6b3dcd3bbab06feeff2cbaf8aade7b6223d6c) Co-authored-by: Nikita Sobolev files: A Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst M Lib/test/test_zoneinfo/test_zoneinfo.py diff --git a/Lib/test/test_zoneinfo/test_zoneinfo.py b/Lib/test/test_zoneinfo/test_zoneinfo.py index a9375fd55857b..59b35ef63f987 100644 --- a/Lib/test/test_zoneinfo/test_zoneinfo.py +++ b/Lib/test/test_zoneinfo/test_zoneinfo.py @@ -1403,44 +1403,50 @@ def tzpath(self): return [self.zoneinfo_data.tzpath] def test_cache_hit(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) - zi_rt = pickle.loads(pkl) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Is non-pickled ZoneInfo"): - self.assertIs(zi_in, zi_rt) + with self.subTest(test="Is non-pickled ZoneInfo"): + self.assertIs(zi_in, zi_rt) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Is unpickled ZoneInfo"): - self.assertIs(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Is unpickled ZoneInfo"): + self.assertIs(zi_rt, zi_rt2) def test_cache_miss(self): - zi_in = self.klass("Europe/Dublin") - pkl = pickle.dumps(zi_in) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_in = self.klass("Europe/Dublin") + pkl = pickle.dumps(zi_in, protocol=proto) - del zi_in - self.klass.clear_cache() # Induce a cache miss - zi_rt = pickle.loads(pkl) - zi_rt2 = pickle.loads(pkl) + del zi_in + self.klass.clear_cache() # Induce a cache miss + zi_rt = pickle.loads(pkl) + zi_rt2 = pickle.loads(pkl) - self.assertIs(zi_rt, zi_rt2) + self.assertIs(zi_rt, zi_rt2) def test_no_cache(self): - zi_no_cache = self.klass.no_cache("Europe/Dublin") + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + zi_no_cache = self.klass.no_cache("Europe/Dublin") - pkl = pickle.dumps(zi_no_cache) - zi_rt = pickle.loads(pkl) + pkl = pickle.dumps(zi_no_cache, protocol=proto) + zi_rt = pickle.loads(pkl) - with self.subTest(test="Not the pickled object"): - self.assertIsNot(zi_rt, zi_no_cache) + with self.subTest(test="Not the pickled object"): + self.assertIsNot(zi_rt, zi_no_cache) - zi_rt2 = pickle.loads(pkl) - with self.subTest(test="Not a second unpickled object"): - self.assertIsNot(zi_rt, zi_rt2) + zi_rt2 = pickle.loads(pkl) + with self.subTest(test="Not a second unpickled object"): + self.assertIsNot(zi_rt, zi_rt2) - zi_cache = self.klass("Europe/Dublin") - with self.subTest(test="Not a cached object"): - self.assertIsNot(zi_rt, zi_cache) + zi_cache = self.klass("Europe/Dublin") + with self.subTest(test="Not a cached object"): + self.assertIsNot(zi_rt, zi_cache) def test_from_file(self): key = "Europe/Dublin" @@ -1456,35 +1462,38 @@ def test_from_file(self): ] for zi, test_name in test_cases: - with self.subTest(test_name=test_name): - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(test_name=test_name, proto=proto): + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi, protocol=proto) def test_pickle_after_from_file(self): # This may be a bit of paranoia, but this test is to ensure that no # global state is maintained in order to handle the pickle cache and # from_file behavior, and that it is possible to interweave the # constructors of each of these and pickling/unpickling without issues. - key = "Europe/Dublin" - zi = self.klass(key) + for proto in range(pickle.HIGHEST_PROTOCOL + 1): + with self.subTest(proto=proto): + key = "Europe/Dublin" + zi = self.klass(key) - pkl_0 = pickle.dumps(zi) - zi_rt_0 = pickle.loads(pkl_0) - self.assertIs(zi, zi_rt_0) + pkl_0 = pickle.dumps(zi, protocol=proto) + zi_rt_0 = pickle.loads(pkl_0) + self.assertIs(zi, zi_rt_0) - with open(self.zoneinfo_data.path_from_key(key), "rb") as f: - zi_ff = self.klass.from_file(f, key=key) + with open(self.zoneinfo_data.path_from_key(key), "rb") as f: + zi_ff = self.klass.from_file(f, key=key) - pkl_1 = pickle.dumps(zi) - zi_rt_1 = pickle.loads(pkl_1) - self.assertIs(zi, zi_rt_1) + pkl_1 = pickle.dumps(zi, protocol=proto) + zi_rt_1 = pickle.loads(pkl_1) + self.assertIs(zi, zi_rt_1) - with self.assertRaises(pickle.PicklingError): - pickle.dumps(zi_ff) + with self.assertRaises(pickle.PicklingError): + pickle.dumps(zi_ff, protocol=proto) - pkl_2 = pickle.dumps(zi) - zi_rt_2 = pickle.loads(pkl_2) - self.assertIs(zi, zi_rt_2) + pkl_2 = pickle.dumps(zi, protocol=proto) + zi_rt_2 = pickle.loads(pkl_2) + self.assertIs(zi, zi_rt_2) class CZoneInfoPickleTest(ZoneInfoPickleTest): diff --git a/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst new file mode 100644 index 0000000000000..fc9783eb9ddaa --- /dev/null +++ b/Misc/NEWS.d/next/Tests/2021-10-22-19-44-13.bpo-45577.dSaNvK.rst @@ -0,0 +1 @@ +Add subtests for all ``pickle`` protocols in ``test_zoneinfo``. From webhook-mailer at python.org Fri Oct 29 04:20:27 2021 From: webhook-mailer at python.org (tjguk) Date: Fri, 29 Oct 2021 08:20:27 -0000 Subject: [Python-checkins] bpo-45621: Small changes to mmap (GH-29247) Message-ID: https://github.com/python/cpython/commit/7bddd96982072d04bd6314da1ee7f40b7f875f00 commit: 7bddd96982072d04bd6314da1ee7f40b7f875f00 branch: main author: Tim Golden committer: tjguk date: 2021-10-29T09:20:21+01:00 summary: bpo-45621: Small changes to mmap (GH-29247) * Small tidy-ups / comments * Use randomized names when testing tagged mmaps to avoid any risk of parallel tests treading on each others' toes files: M Lib/test/test_mmap.py M Modules/mmapmodule.c diff --git a/Lib/test/test_mmap.py b/Lib/test/test_mmap.py index 82e2d2adb7dcf..014171cbb4911 100644 --- a/Lib/test/test_mmap.py +++ b/Lib/test/test_mmap.py @@ -7,6 +7,7 @@ import itertools import random import socket +import string import sys import weakref @@ -15,6 +16,10 @@ PAGESIZE = mmap.PAGESIZE +tagname_prefix = f'python_{os.getpid()}_test_mmap' +def random_tagname(length=10): + suffix = ''.join(random.choices(string.ascii_uppercase, k=length)) + return f'{tagname_prefix}_{suffix}' class MmapTests(unittest.TestCase): @@ -610,11 +615,13 @@ def test_tagname(self): data1 = b"0123456789" data2 = b"abcdefghij" assert len(data1) == len(data2) + tagname1 = random_tagname() + tagname2 = random_tagname() # Test same tag - m1 = mmap.mmap(-1, len(data1), tagname="foo") + m1 = mmap.mmap(-1, len(data1), tagname=tagname1) m1[:] = data1 - m2 = mmap.mmap(-1, len(data2), tagname="foo") + m2 = mmap.mmap(-1, len(data2), tagname=tagname1) m2[:] = data2 self.assertEqual(m1[:], data2) self.assertEqual(m2[:], data2) @@ -622,9 +629,9 @@ def test_tagname(self): m1.close() # Test different tag - m1 = mmap.mmap(-1, len(data1), tagname="foo") + m1 = mmap.mmap(-1, len(data1), tagname=tagname1) m1[:] = data1 - m2 = mmap.mmap(-1, len(data2), tagname="boo") + m2 = mmap.mmap(-1, len(data2), tagname=tagname2) m2[:] = data2 self.assertEqual(m1[:], data1) self.assertEqual(m2[:], data2) @@ -635,7 +642,7 @@ def test_tagname(self): @unittest.skipUnless(os.name == 'nt', 'requires Windows') def test_sizeof(self): m1 = mmap.mmap(-1, 100) - tagname = "foo" + tagname = random_tagname() m2 = mmap.mmap(-1, 100, tagname=tagname) self.assertEqual(sys.getsizeof(m2), sys.getsizeof(m1) + len(tagname) + 1) @@ -643,9 +650,10 @@ def test_sizeof(self): @unittest.skipUnless(os.name == 'nt', 'requires Windows') def test_crasher_on_windows(self): # Should not crash (Issue 1733986) - m = mmap.mmap(-1, 1000, tagname="foo") + tagname = random_tagname() + m = mmap.mmap(-1, 1000, tagname=tagname) try: - mmap.mmap(-1, 5000, tagname="foo")[:] # same tagname, but larger size + mmap.mmap(-1, 5000, tagname=tagname)[:] # same tagname, but larger size except: pass m.close() @@ -857,7 +865,7 @@ def test_resize_succeeds_with_error_for_second_named_mapping(self): """ start_size = 2 * PAGESIZE reduced_size = PAGESIZE - tagname = "TEST" + tagname = random_tagname() data_length = 8 data = bytes(random.getrandbits(8) for _ in range(data_length)) diff --git a/Modules/mmapmodule.c b/Modules/mmapmodule.c index dfa10f633bbd5..21d53365776ab 100644 --- a/Modules/mmapmodule.c +++ b/Modules/mmapmodule.c @@ -32,7 +32,6 @@ #ifdef MS_WINDOWS #include -#include static int my_getpagesize(void) { @@ -505,21 +504,6 @@ mmap_resize_method(mmap_object *self, } { - /* - To resize an mmap on Windows: - - - Close the existing mapping - - If the mapping is backed to a named file: - unmap the view, clear the data, and resize the file - If the file can't be resized (eg because it has other mapped references - to it) then let the mapping be recreated at the original size and set - an error code so an exception will be raised. - - Create a new mapping of the relevant size to the same file - - Map a new view of the resized file - - If the mapping is backed by the pagefile: - copy any previous data into the new mapped area - unmap the original view which will release the memory - */ #ifdef MS_WINDOWS DWORD error = 0, file_resize_error = 0; char* old_data = self->data; @@ -567,6 +551,11 @@ mmap_resize_method(mmap_object *self, self->tagname); error = GetLastError(); + /* ERROR_ALREADY_EXISTS implies that between our closing the handle above and + calling CreateFileMapping here, someone's created a different mapping with + the same name. There's nothing we can usefully do so we invalidate our + mapping and error out. + */ if (error == ERROR_ALREADY_EXISTS) { CloseHandle(self->map_handle); self->map_handle = NULL; From webhook-mailer at python.org Fri Oct 29 05:40:27 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 29 Oct 2021 09:40:27 -0000 Subject: [Python-checkins] [3.10] bpo-45502: Fix test_shelve (GH-29003) (GH-29305) Message-ID: https://github.com/python/cpython/commit/6b867022d926be9fcc6f8038fb1093ba8c348ca5 commit: 6b867022d926be9fcc6f8038fb1093ba8c348ca5 branch: 3.10 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-29T12:40:22+03:00 summary: [3.10] bpo-45502: Fix test_shelve (GH-29003) (GH-29305) Run test_shelve with all underlying dbm implementations and pickle protocols. Also make test_shelve discoverable. (cherry picked from commit b781cc3bfce7c052728b06aad9f1a467cced289d) files: M Lib/test/test_shelve.py diff --git a/Lib/test/test_shelve.py b/Lib/test/test_shelve.py index cfdd67c26c5f5..dcb5aefc3bae1 100644 --- a/Lib/test/test_shelve.py +++ b/Lib/test/test_shelve.py @@ -1,4 +1,6 @@ import unittest +import dbm +import os import shelve import glob import pickle @@ -43,12 +45,8 @@ def copy(self): class TestCase(unittest.TestCase): - - fn = "shelftemp.db" - - def tearDown(self): - for f in glob.glob(self.fn+"*"): - os_helper.unlink(f) + dirname = os_helper.TESTFN + fn = os.path.join(os_helper.TESTFN, "shelftemp.db") def test_close(self): d1 = {} @@ -65,29 +63,24 @@ def test_close(self): else: self.fail('Closed shelf should not find a key') - def test_ascii_file_shelf(self): - s = shelve.open(self.fn, protocol=0) + def test_open_template(self, protocol=None): + os.mkdir(self.dirname) + self.addCleanup(os_helper.rmtree, self.dirname) + s = shelve.open(self.fn, protocol=protocol) try: s['key1'] = (1,2,3,4) self.assertEqual(s['key1'], (1,2,3,4)) finally: s.close() + def test_ascii_file_shelf(self): + self.test_open_template(protocol=0) + def test_binary_file_shelf(self): - s = shelve.open(self.fn, protocol=1) - try: - s['key1'] = (1,2,3,4) - self.assertEqual(s['key1'], (1,2,3,4)) - finally: - s.close() + self.test_open_template(protocol=1) def test_proto2_file_shelf(self): - s = shelve.open(self.fn, protocol=2) - try: - s['key1'] = (1,2,3,4) - self.assertEqual(s['key1'], (1,2,3,4)) - finally: - s.close() + self.test_open_template(protocol=2) def test_in_memory_shelf(self): d1 = byteskeydict() @@ -164,63 +157,52 @@ def test_default_protocol(self): with shelve.Shelf({}) as s: self.assertEqual(s._protocol, pickle.DEFAULT_PROTOCOL) -from test import mapping_tests -class TestShelveBase(mapping_tests.BasicTestMappingProtocol): - fn = "shelftemp.db" - counter = 0 - def __init__(self, *args, **kw): - self._db = [] - mapping_tests.BasicTestMappingProtocol.__init__(self, *args, **kw) +class TestShelveBase: type2test = shelve.Shelf + def _reference(self): return {"key1":"value1", "key2":2, "key3":(1,2,3)} + + +class TestShelveInMemBase(TestShelveBase): def _empty_mapping(self): - if self._in_mem: - x= shelve.Shelf(byteskeydict(), **self._args) - else: - self.counter+=1 - x= shelve.open(self.fn+str(self.counter), **self._args) - self._db.append(x) + return shelve.Shelf(byteskeydict(), **self._args) + + +class TestShelveFileBase(TestShelveBase): + counter = 0 + + def _empty_mapping(self): + self.counter += 1 + x = shelve.open(self.base_path + str(self.counter), **self._args) + self.addCleanup(x.close) return x - def tearDown(self): - for db in self._db: - db.close() - self._db = [] - if not self._in_mem: - for f in glob.glob(self.fn+"*"): - os_helper.unlink(f) - -class TestAsciiFileShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = False -class TestBinaryFileShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = False -class TestProto2FileShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = False -class TestAsciiMemShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = True -class TestBinaryMemShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = True -class TestProto2MemShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = True - -def test_main(): - for module in dbm_iterator(): - support.run_unittest( - TestAsciiFileShelve, - TestBinaryFileShelve, - TestProto2FileShelve, - TestAsciiMemShelve, - TestBinaryMemShelve, - TestProto2MemShelve, - TestCase - ) + + def setUp(self): + dirname = os_helper.TESTFN + os.mkdir(dirname) + self.addCleanup(os_helper.rmtree, dirname) + self.base_path = os.path.join(dirname, "shelftemp.db") + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) + dbm._defaultmod = self.dbm_mod + + +from test import mapping_tests + +for proto in range(pickle.HIGHEST_PROTOCOL + 1): + bases = (TestShelveInMemBase, mapping_tests.BasicTestMappingProtocol) + name = f'TestProto{proto}MemShelve' + globals()[name] = type(name, bases, + {'_args': {'protocol': proto}}) + bases = (TestShelveFileBase, mapping_tests.BasicTestMappingProtocol) + for dbm_mod in dbm_iterator(): + assert dbm_mod.__name__.startswith('dbm.') + suffix = dbm_mod.__name__[4:] + name = f'TestProto{proto}File_{suffix}Shelve' + globals()[name] = type(name, bases, + {'dbm_mod': dbm_mod, '_args': {'protocol': proto}}) + if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 29 06:04:02 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Fri, 29 Oct 2021 10:04:02 -0000 Subject: [Python-checkins] [3.10] bo-45655: Add "relevant PEPs" section to typing documentation (GH-29302) Message-ID: https://github.com/python/cpython/commit/2c8a0027e5555e371c1293f26b3262000b8cfe8a commit: 2c8a0027e5555e371c1293f26b3262000b8cfe8a branch: 3.10 author: Alex Waygood committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-29T18:03:48+08:00 summary: [3.10] bo-45655: Add "relevant PEPs" section to typing documentation (GH-29302) The list of PEPs at the top of the documentation for the ``typing`` module has become too long to be readable. This PR proposes presenting this information in a more structured and readable way by adding a new "relevant PEPs" section to the ``typing`` docs. (cherry picked from commit 03db1bbfd2d3f5a343c293b2f0e09a1e962df7ea) Co-authored-by: Alex Waygood Co-authored-by: ?ukasz Langa files: A Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst M Doc/library/typing.rst M Doc/tools/susp-ignored.csv diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index c402c73b487ea..e5b0468399e50 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -17,13 +17,11 @@ -------------- -This module provides runtime support for type hints as specified by -:pep:`484`, :pep:`526`, :pep:`544`, :pep:`586`, :pep:`589`, :pep:`591`, -:pep:`593`, :pep:`612`, :pep:`613` and :pep:`647`. -The most fundamental support consists of the types :data:`Any`, :data:`Union`, -:data:`Tuple`, :data:`Callable`, :class:`TypeVar`, and -:class:`Generic`. For full specification please see :pep:`484`. For -a simplified introduction to type hints see :pep:`483`. +This module provides runtime support for type hints. The most fundamental +support consists of the types :data:`Any`, :data:`Union`, :data:`Tuple`, +:data:`Callable`, :class:`TypeVar`, and :class:`Generic`. For a full +specification, please see :pep:`484`. For a simplified introduction to type +hints, see :pep:`483`. The function below takes and returns a string and is annotated as follows:: @@ -35,6 +33,42 @@ In the function ``greeting``, the argument ``name`` is expected to be of type :class:`str` and the return type :class:`str`. Subtypes are accepted as arguments. +.. _relevant-peps: + +Relevant PEPs +============= + +Since the initial introduction of type hints in :pep:`484` and :pep:`483`, a +number of PEPs have modified and enhanced Python's framework for type +annotations. These include: + +* :pep:`526`: Syntax for Variable Annotations + *Introducing* syntax for annotating variables outside of function + definitions, and :data:`ClassVar` +* :pep:`544`: Protocols: Structural subtyping (static duck typing) + *Introducing* :class:`Protocol` and the + :func:`@runtime_checkable` decorator +* :pep:`585`: Type Hinting Generics In Standard Collections + *Introducing* the ability to use builtin collections and ABCs as + :term:`generic types` +* :pep:`586`: Literal Types + *Introducing* :data:`Literal` +* :pep:`589`: TypedDict: Type Hints for Dictionaries with a Fixed Set of Keys + *Introducing* :class:`TypedDict` +* :pep:`591`: Adding a final qualifier to typing + *Introducing* :data:`Final` and the :func:`@final` decorator +* :pep:`593`: Flexible function and variable annotations + *Introducing* :data:`Annotated` +* :pep:`604`: Allow writing union types as ``X | Y`` + *Introducing* :data:`types.UnionType` and the ability to use + the binary-or operator ``|`` as syntactic sugar for a union of types +* :pep:`612`: Parameter Specification Variables + *Introducing* :class:`ParamSpec` and :data:`Concatenate` +* :pep:`613`: Explicit Type Aliases + *Introducing* :data:`TypeAlias` +* :pep:`647`: User-Defined Type Guards + *Introducing* :data:`TypeGuard` + .. _type-aliases: Type aliases diff --git a/Doc/tools/susp-ignored.csv b/Doc/tools/susp-ignored.csv index 54360c2f8ebaa..9cbd09ad20357 100644 --- a/Doc/tools/susp-ignored.csv +++ b/Doc/tools/susp-ignored.csv @@ -376,7 +376,7 @@ library/importlib.metadata,,:main,"EntryPoint(name='wheel', value='wheel.cli:mai library/importlib.metadata,,`,loading the metadata for packages for the indicated ``context``. library/re,,`,"`" using/configure,84,:db2,=db1:db2:... -library/typing,1011,`,# Type of ``val`` is narrowed to ``str`` -library/typing,1011,`,"# Else, type of ``val`` is narrowed to ``float``." -library/typing,1011,`,# Type of ``val`` is narrowed to ``List[str]``. -library/typing,1011,`,# Type of ``val`` remains as ``List[object]``. +library/typing,,`,# Type of ``val`` is narrowed to ``str`` +library/typing,,`,"# Else, type of ``val`` is narrowed to ``float``." +library/typing,,`,# Type of ``val`` is narrowed to ``List[str]``. +library/typing,,`,# Type of ``val`` remains as ``List[object]``. diff --git a/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst b/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst new file mode 100644 index 0000000000000..fc5b3d0788817 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-28-19-22-55.bpo-45655.aPYGaS.rst @@ -0,0 +1,2 @@ +Add a new "relevant PEPs" section to the top of the documentation for the +``typing`` module. Patch by Alex Waygood. From webhook-mailer at python.org Fri Oct 29 06:18:39 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Fri, 29 Oct 2021 10:18:39 -0000 Subject: [Python-checkins] [3.9] [3.10] bpo-45502: Fix test_shelve (GH-29003) (GH-29305) (GH-29306) Message-ID: https://github.com/python/cpython/commit/a043706f907e82ee6a562005991ff0b896a4e64d commit: a043706f907e82ee6a562005991ff0b896a4e64d branch: 3.9 author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-29T13:18:34+03:00 summary: [3.9] [3.10] bpo-45502: Fix test_shelve (GH-29003) (GH-29305) (GH-29306) Run test_shelve with all underlying dbm implementations and pickle protocols. Also make test_shelve discoverable. (cherry picked from commit b781cc3bfce7c052728b06aad9f1a467cced289d). (cherry picked from commit 6b867022d926be9fcc6f8038fb1093ba8c348ca5) files: M Lib/test/test_shelve.py diff --git a/Lib/test/test_shelve.py b/Lib/test/test_shelve.py index 9ffe2cbeae4d8..61dae75cb325e 100644 --- a/Lib/test/test_shelve.py +++ b/Lib/test/test_shelve.py @@ -1,6 +1,9 @@ import unittest +import dbm +import os import shelve import glob +import pickle from test import support from collections.abc import MutableMapping from test.test_dbm import dbm_iterator @@ -40,12 +43,8 @@ def copy(self): class TestCase(unittest.TestCase): - - fn = "shelftemp.db" - - def tearDown(self): - for f in glob.glob(self.fn+"*"): - support.unlink(f) + dirname = support.TESTFN + fn = os.path.join(support.TESTFN, "shelftemp.db") def test_close(self): d1 = {} @@ -62,29 +61,24 @@ def test_close(self): else: self.fail('Closed shelf should not find a key') - def test_ascii_file_shelf(self): - s = shelve.open(self.fn, protocol=0) + def test_open_template(self, protocol=None): + os.mkdir(self.dirname) + self.addCleanup(support.rmtree, self.dirname) + s = shelve.open(self.fn, protocol=protocol) try: s['key1'] = (1,2,3,4) self.assertEqual(s['key1'], (1,2,3,4)) finally: s.close() + def test_ascii_file_shelf(self): + self.test_open_template(protocol=0) + def test_binary_file_shelf(self): - s = shelve.open(self.fn, protocol=1) - try: - s['key1'] = (1,2,3,4) - self.assertEqual(s['key1'], (1,2,3,4)) - finally: - s.close() + self.test_open_template(protocol=1) def test_proto2_file_shelf(self): - s = shelve.open(self.fn, protocol=2) - try: - s['key1'] = (1,2,3,4) - self.assertEqual(s['key1'], (1,2,3,4)) - finally: - s.close() + self.test_open_template(protocol=2) def test_in_memory_shelf(self): d1 = byteskeydict() @@ -161,63 +155,52 @@ def test_default_protocol(self): with shelve.Shelf({}) as s: self.assertEqual(s._protocol, 3) -from test import mapping_tests -class TestShelveBase(mapping_tests.BasicTestMappingProtocol): - fn = "shelftemp.db" - counter = 0 - def __init__(self, *args, **kw): - self._db = [] - mapping_tests.BasicTestMappingProtocol.__init__(self, *args, **kw) +class TestShelveBase: type2test = shelve.Shelf + def _reference(self): return {"key1":"value1", "key2":2, "key3":(1,2,3)} + + +class TestShelveInMemBase(TestShelveBase): def _empty_mapping(self): - if self._in_mem: - x= shelve.Shelf(byteskeydict(), **self._args) - else: - self.counter+=1 - x= shelve.open(self.fn+str(self.counter), **self._args) - self._db.append(x) + return shelve.Shelf(byteskeydict(), **self._args) + + +class TestShelveFileBase(TestShelveBase): + counter = 0 + + def _empty_mapping(self): + self.counter += 1 + x = shelve.open(self.base_path + str(self.counter), **self._args) + self.addCleanup(x.close) return x - def tearDown(self): - for db in self._db: - db.close() - self._db = [] - if not self._in_mem: - for f in glob.glob(self.fn+"*"): - support.unlink(f) - -class TestAsciiFileShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = False -class TestBinaryFileShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = False -class TestProto2FileShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = False -class TestAsciiMemShelve(TestShelveBase): - _args={'protocol':0} - _in_mem = True -class TestBinaryMemShelve(TestShelveBase): - _args={'protocol':1} - _in_mem = True -class TestProto2MemShelve(TestShelveBase): - _args={'protocol':2} - _in_mem = True - -def test_main(): - for module in dbm_iterator(): - support.run_unittest( - TestAsciiFileShelve, - TestBinaryFileShelve, - TestProto2FileShelve, - TestAsciiMemShelve, - TestBinaryMemShelve, - TestProto2MemShelve, - TestCase - ) + + def setUp(self): + dirname = support.TESTFN + os.mkdir(dirname) + self.addCleanup(support.rmtree, dirname) + self.base_path = os.path.join(dirname, "shelftemp.db") + self.addCleanup(setattr, dbm, '_defaultmod', dbm._defaultmod) + dbm._defaultmod = self.dbm_mod + + +from test import mapping_tests + +for proto in range(pickle.HIGHEST_PROTOCOL + 1): + bases = (TestShelveInMemBase, mapping_tests.BasicTestMappingProtocol) + name = f'TestProto{proto}MemShelve' + globals()[name] = type(name, bases, + {'_args': {'protocol': proto}}) + bases = (TestShelveFileBase, mapping_tests.BasicTestMappingProtocol) + for dbm_mod in dbm_iterator(): + assert dbm_mod.__name__.startswith('dbm.') + suffix = dbm_mod.__name__[4:] + name = f'TestProto{proto}File_{suffix}Shelve' + globals()[name] = type(name, bases, + {'dbm_mod': dbm_mod, '_args': {'protocol': proto}}) + if __name__ == "__main__": - test_main() + unittest.main() From webhook-mailer at python.org Fri Oct 29 09:40:49 2021 From: webhook-mailer at python.org (vsajip) Date: Fri, 29 Oct 2021 13:40:49 -0000 Subject: [Python-checkins] bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) Message-ID: https://github.com/python/cpython/commit/8a77f59de51f1fd6062f0fefe73ee3059d714144 commit: 8a77f59de51f1fd6062f0fefe73ee3059d714144 branch: main author: Vinay Sajip committer: vsajip date: 2021-10-29T14:40:37+01:00 summary: bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) files: M Lib/logging/handlers.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index b613bec1c4270..d42c48de5f062 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -371,8 +371,13 @@ def getFilesToDelete(self): for fileName in fileNames: if fileName[:plen] == prefix: suffix = fileName[plen:] - if self.extMatch.match(suffix): - result.append(os.path.join(dirName, fileName)) + # See bpo-45628: The date/time suffix could be anywhere in the + # filename + parts = suffix.split('.') + for part in parts: + if self.extMatch.match(part): + result.append(os.path.join(dirName, fileName)) + break if len(result) < self.backupCount: result = [] else: From webhook-mailer at python.org Fri Oct 29 09:56:31 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Fri, 29 Oct 2021 13:56:31 -0000 Subject: [Python-checkins] bpo-45655: Add ref to union type expressions at top of typing docs (GH-29309) Message-ID: https://github.com/python/cpython/commit/d9575218d7ab3d85b15ce3d4779660b9b724d343 commit: d9575218d7ab3d85b15ce3d4779660b9b724d343 branch: main author: Alex Waygood committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-29T21:56:21+08:00 summary: bpo-45655: Add ref to union type expressions at top of typing docs (GH-29309) files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index 6f501ec136ee6..eb95af378d45f 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -61,7 +61,8 @@ annotations. These include: *Introducing* :data:`Annotated` * :pep:`604`: Allow writing union types as ``X | Y`` *Introducing* :data:`types.UnionType` and the ability to use - the binary-or operator ``|`` as syntactic sugar for a union of types + the binary-or operator ``|`` to signify a + :ref:`union of types` * :pep:`612`: Parameter Specification Variables *Introducing* :class:`ParamSpec` and :data:`Concatenate` * :pep:`613`: Explicit Type Aliases From webhook-mailer at python.org Fri Oct 29 10:16:28 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Oct 2021 14:16:28 -0000 Subject: [Python-checkins] bpo-45655: Add ref to union type expressions at top of typing docs (GH-29309) Message-ID: https://github.com/python/cpython/commit/6742b0dfb61ebdb92a1ff633ec071734b5d39981 commit: 6742b0dfb61ebdb92a1ff633ec071734b5d39981 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-29T07:16:19-07:00 summary: bpo-45655: Add ref to union type expressions at top of typing docs (GH-29309) (cherry picked from commit d9575218d7ab3d85b15ce3d4779660b9b724d343) Co-authored-by: Alex Waygood files: M Doc/library/typing.rst diff --git a/Doc/library/typing.rst b/Doc/library/typing.rst index e5b0468399e50..7652c87827220 100644 --- a/Doc/library/typing.rst +++ b/Doc/library/typing.rst @@ -61,7 +61,8 @@ annotations. These include: *Introducing* :data:`Annotated` * :pep:`604`: Allow writing union types as ``X | Y`` *Introducing* :data:`types.UnionType` and the ability to use - the binary-or operator ``|`` as syntactic sugar for a union of types + the binary-or operator ``|`` to signify a + :ref:`union of types` * :pep:`612`: Parameter Specification Variables *Introducing* :class:`ParamSpec` and :data:`Concatenate` * :pep:`613`: Explicit Type Aliases From webhook-mailer at python.org Fri Oct 29 11:24:50 2021 From: webhook-mailer at python.org (vsajip) Date: Fri, 29 Oct 2021 15:24:50 -0000 Subject: [Python-checkins] [3.9] bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) (GH-29313) Message-ID: https://github.com/python/cpython/commit/317e0c99e3804310f4bee23e497d9d84b717d7f7 commit: 317e0c99e3804310f4bee23e497d9d84b717d7f7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vsajip date: 2021-10-29T16:24:41+01:00 summary: [3.9] bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) (GH-29313) files: M Lib/logging/handlers.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 7f1f10551c304..f8040c498c10d 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -368,8 +368,13 @@ def getFilesToDelete(self): for fileName in fileNames: if fileName[:plen] == prefix: suffix = fileName[plen:] - if self.extMatch.match(suffix): - result.append(os.path.join(dirName, fileName)) + # See bpo-45628: The date/time suffix could be anywhere in the + # filename + parts = suffix.split('.') + for part in parts: + if self.extMatch.match(part): + result.append(os.path.join(dirName, fileName)) + break if len(result) < self.backupCount: result = [] else: From webhook-mailer at python.org Fri Oct 29 11:25:36 2021 From: webhook-mailer at python.org (vsajip) Date: Fri, 29 Oct 2021 15:25:36 -0000 Subject: [Python-checkins] [3.10] bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) (GH-29314) Message-ID: https://github.com/python/cpython/commit/191a93905a84f272b2232701dc5dcc69987330f5 commit: 191a93905a84f272b2232701dc5dcc69987330f5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vsajip date: 2021-10-29T16:25:31+01:00 summary: [3.10] bpo-45628: Check all parts of the suffix for an extension match. (GH-29310) (GH-29314) files: M Lib/logging/handlers.py diff --git a/Lib/logging/handlers.py b/Lib/logging/handlers.py index 4dcbe4530fcfc..4e8f0a8cc2f46 100644 --- a/Lib/logging/handlers.py +++ b/Lib/logging/handlers.py @@ -371,8 +371,13 @@ def getFilesToDelete(self): for fileName in fileNames: if fileName[:plen] == prefix: suffix = fileName[plen:] - if self.extMatch.match(suffix): - result.append(os.path.join(dirName, fileName)) + # See bpo-45628: The date/time suffix could be anywhere in the + # filename + parts = suffix.split('.') + for part in parts: + if self.extMatch.match(part): + result.append(os.path.join(dirName, fileName)) + break if len(result) < self.backupCount: result = [] else: From webhook-mailer at python.org Fri Oct 29 11:50:06 2021 From: webhook-mailer at python.org (tiran) Date: Fri, 29 Oct 2021 15:50:06 -0000 Subject: [Python-checkins] bpo-45548: Some test modules must be built as shared libs (GH-29268) Message-ID: https://github.com/python/cpython/commit/f0150ac94a85c863ec1dcb58b9e33ed7ce465ec8 commit: f0150ac94a85c863ec1dcb58b9e33ed7ce465ec8 branch: main author: Christian Heimes committer: tiran date: 2021-10-29T17:49:57+02:00 summary: bpo-45548: Some test modules must be built as shared libs (GH-29268) Some test cases don't work when test modules are static extensions. Add dependency on Modules/config.c to trigger a rebuild whenever a module build type is changed. ``makesetup`` puts shared extensions into ``Modules/`` directory. Create symlinks from pybuilddir so the extensions can be imported. Note: It is not possible to use the content of pybuilddir.txt as a build target. Makefile evaluates target variables in the first pass. The pybuilddir.txt file does not exist at that point. files: A Misc/NEWS.d/next/Build/2021-10-28-14-47-22.bpo-45548.mdCBxB.rst M Doc/whatsnew/3.11.rst M Makefile.pre.in M Modules/Setup M Modules/makesetup M setup.py diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 21ad4669476ef..156bfbd9109ee 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -507,6 +507,13 @@ Build Changes except empty tuple singleton. (Contributed by Christian Heimes in :issue:`45522`) +* ``Modules/Setup`` and ``Modules/makesetup`` have been improved and tied up. + Extension modules can now be built through ``makesetup``. All except some + test modules can be linked statically into main binary or library. + (Contributed by Brett Cannon and Christian Heimes in :issue:`45548`, + :issue:`45570`, :issue:`45571`, and :issue:`43974`.) + + C API Changes ============= diff --git a/Makefile.pre.in b/Makefile.pre.in index 60acc16beeee3..322800ae26e4c 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -705,8 +705,17 @@ $(DLLLIBRARY) libpython$(LDVERSION).dll.a: $(LIBRARY_OBJS) fi -oldsharedmods: $(SHAREDMODS) - +# create relative links from build/lib.platform/egg.so to Modules/egg.so +# pybuilddir.txt is created too late. We cannot use it in Makefile +# targets. ln --relative is not portable. +oldsharedmods: $(SHAREDMODS) pybuilddir.txt + @target=`cat pybuilddir.txt`; \ + $(MKDIR_P) $$target; \ + for mod in X $(SHAREDMODS); do \ + if test $$mod != X; then \ + $(LN) -sf ../../$$mod $$target/`basename $$mod`; \ + fi; \ + done Makefile Modules/config.c: Makefile.pre \ $(srcdir)/Modules/config.c.in \ diff --git a/Misc/NEWS.d/next/Build/2021-10-28-14-47-22.bpo-45548.mdCBxB.rst b/Misc/NEWS.d/next/Build/2021-10-28-14-47-22.bpo-45548.mdCBxB.rst new file mode 100644 index 0000000000000..e6ccd52274de5 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-28-14-47-22.bpo-45548.mdCBxB.rst @@ -0,0 +1,4 @@ +``Modules/Setup`` and ``Modules/makesetup`` have been improved. The +``Setup`` file now contains working rules for all extensions. Outdated +comments have been removed. Rules defined by ``makesetup`` track +dependencies correctly. diff --git a/Modules/Setup b/Modules/Setup index b4eae38fdff60..57584f38fd8e7 100644 --- a/Modules/Setup +++ b/Modules/Setup @@ -297,14 +297,16 @@ xxsubtype xxsubtype.c # Required for the test suite to pass! #_xxsubinterpreters _xxsubinterpretersmodule.c #_xxtestfuzz _xxtestfuzz/_xxtestfuzz.c _xxtestfuzz/fuzzer.c -#_ctypes_test _ctypes/_ctypes_test.c #_testbuffer _testbuffer.c -#_testimportmultiple _testimportmultiple.c #_testinternalcapi _testinternalcapi.c -#_testmultiphase _testmultiphase.c + +# Some testing modules MUST be built as shared libraries. #*shared* -#_testcapi _testcapimodule.c # CANNOT be statically compiled! +#_ctypes_test _ctypes/_ctypes_test.c +#_testcapi _testcapimodule.c +#_testimportmultiple _testimportmultiple.c +#_testmultiphase _testmultiphase.c # --- # Uncommenting the following line tells makesetup that all following modules diff --git a/Modules/makesetup b/Modules/makesetup index 7547315281643..543992c4a0294 100755 --- a/Modules/makesetup +++ b/Modules/makesetup @@ -241,7 +241,8 @@ sed -e 's/[ ]*#.*//' -e '/^[ ]*$/d' | cc="$cc \$(PY_BUILTIN_MODULE_CFLAGS)";; esac mods_upper=$(echo $mods | tr '[a-z]' '[A-Z]') - rule="$obj: $src \$(MODULE_${mods_upper}_DEPS) \$(PYTHON_HEADERS); $cc $cpps -c $src -o $obj" + # force rebuild when header file or module build flavor (static/shared) is changed + rule="$obj: $src \$(MODULE_${mods_upper}_DEPS) \$(PYTHON_HEADERS) Modules/config.c; $cc $cpps -c $src -o $obj" echo "$rule" >>$rulesf done case $doconfig in diff --git a/setup.py b/setup.py index f32dd4c6e5e14..6fa8a674eda31 100644 --- a/setup.py +++ b/setup.py @@ -426,12 +426,13 @@ def update_sources_depends(self): # re-compile extensions if a header file has been changed ext.depends.extend(headers) - def remove_configured_extensions(self): + def handle_configured_extensions(self): # The sysconfig variables built by makesetup that list the already # built modules and the disabled modules as configured by the Setup # files. - sysconf_built = sysconfig.get_config_var('MODBUILT_NAMES').split() - sysconf_dis = sysconfig.get_config_var('MODDISABLED_NAMES').split() + sysconf_built = set(sysconfig.get_config_var('MODBUILT_NAMES').split()) + sysconf_shared = set(sysconfig.get_config_var('MODSHARED_NAMES').split()) + sysconf_dis = set(sysconfig.get_config_var('MODDISABLED_NAMES').split()) mods_built = [] mods_disabled = [] @@ -449,11 +450,15 @@ def remove_configured_extensions(self): mods_configured] # Remove the shared libraries built by a previous build. for ext in mods_configured: + # Don't remove shared extensions which have been built + # by Modules/Setup + if ext.name in sysconf_shared: + continue fullpath = self.get_ext_fullpath(ext.name) - if os.path.exists(fullpath): + if os.path.lexists(fullpath): os.unlink(fullpath) - return (mods_built, mods_disabled) + return mods_built, mods_disabled def set_compiler_executables(self): # When you run "make CC=altcc" or something similar, you really want @@ -478,7 +483,7 @@ def build_extensions(self): self.remove_disabled() self.update_sources_depends() - mods_built, mods_disabled = self.remove_configured_extensions() + mods_built, mods_disabled = self.handle_configured_extensions() self.set_compiler_executables() if LIST_MODULE_NAMES: From webhook-mailer at python.org Fri Oct 29 13:21:24 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Oct 2021 17:21:24 -0000 Subject: [Python-checkins] bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270) Message-ID: https://github.com/python/cpython/commit/d8ca47c943c4ad396a9c70aff35bbbf4b1868876 commit: d8ca47c943c4ad396a9c70aff35bbbf4b1868876 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-29T10:21:15-07:00 summary: bpo-45562: Ensure all tokenizer debug messages are printed to stderr (GH-29270) (cherry picked from commit cdc7a5827754bec83970bb052d410d55f85b3fff) Co-authored-by: Pablo Galindo Salgado files: M Parser/tokenizer.c diff --git a/Parser/tokenizer.c b/Parser/tokenizer.c index 6563cdfd7cc60..519300f3731f0 100644 --- a/Parser/tokenizer.c +++ b/Parser/tokenizer.c @@ -1047,7 +1047,7 @@ tok_nextc(struct tok_state *tok) #if defined(Py_DEBUG) if (Py_DebugFlag) { fprintf(stderr, "line[%d] = ", tok->lineno); - print_escape(stdout, tok->cur, tok->inp - tok->cur); + print_escape(stderr, tok->cur, tok->inp - tok->cur); fprintf(stderr, " tok->done = %d\n", tok->done); } #endif From webhook-mailer at python.org Fri Oct 29 16:17:31 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:17:31 -0000 Subject: [Python-checkins] bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) Message-ID: https://github.com/python/cpython/commit/3877fc02f7a8801ba5ce0e94b6075b3fdd9778d0 commit: 3877fc02f7a8801ba5ce0e94b6075b3fdd9778d0 branch: main author: Ian Fisher committer: ambv date: 2021-10-29T22:17:22+02:00 summary: bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index fe1b64ade9561..19a4155542382 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1095,6 +1095,12 @@ If a timestamp stored in SQLite has a fractional part longer than 6 numbers, its value will be truncated to microsecond precision by the timestamp converter. +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. .. _sqlite3-controlling-transactions: From webhook-mailer at python.org Fri Oct 29 16:18:50 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:18:50 -0000 Subject: [Python-checkins] bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) Message-ID: https://github.com/python/cpython/commit/b17cfd150f4dc2816975d304a71110a2d445eaf0 commit: b17cfd150f4dc2816975d304a71110a2d445eaf0 branch: main author: andrei kulakov committer: ambv date: 2021-10-29T22:18:45+02:00 summary: bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) Co-authored-by: ?ukasz Langa files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 3bfefc6bcd57d..8092397be650f 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -170,9 +170,10 @@ process and user. .. data:: environ - A :term:`mapping` object representing the string environment. For example, - ``environ['HOME']`` is the pathname of your home directory (on some platforms), - and is equivalent to ``getenv("HOME")`` in C. + A :term:`mapping` object where keys and values are strings that represent + the process environment. For example, ``environ['HOME']`` is the pathname + of your home directory (on some platforms), and is equivalent to + ``getenv("HOME")`` in C. This mapping is captured the first time the :mod:`os` module is imported, typically during Python startup as part of processing :file:`site.py`. Changes @@ -209,10 +210,10 @@ process and user. .. data:: environb - Bytes version of :data:`environ`: a :term:`mapping` object representing the - environment as byte strings. :data:`environ` and :data:`environb` are - synchronized (modify :data:`environb` updates :data:`environ`, and vice - versa). + Bytes version of :data:`environ`: a :term:`mapping` object where both keys + and values are :class:`bytes` objects representing the process environment. + :data:`environ` and :data:`environb` are synchronized (modifying + :data:`environb` updates :data:`environ`, and vice versa). :data:`environb` is only available if :data:`supports_bytes_environ` is ``True``. From webhook-mailer at python.org Fri Oct 29 16:22:07 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:22:07 -0000 Subject: [Python-checkins] bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) Message-ID: https://github.com/python/cpython/commit/e2e62b3808691e15fa44b883270023e42dcad958 commit: e2e62b3808691e15fa44b883270023e42dcad958 branch: main author: Erlend Egeberg Aasland committer: ambv date: 2021-10-29T22:21:58+02:00 summary: bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) files: A Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst M Modules/_sqlite/connection.c diff --git a/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst new file mode 100644 index 0000000000000..13a3b237434eb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst @@ -0,0 +1,2 @@ +:meth:`sqlite3.connect` now correctly raises :exc:`MemoryError` if the +underlying SQLite API signals memory error. Patch by Erlend E. Aasland. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index da2f12e8f99c2..94c38ad395440 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -165,6 +165,10 @@ pysqlite_connection_init_impl(pysqlite_Connection *self, (uri ? SQLITE_OPEN_URI : 0), NULL); Py_END_ALLOW_THREADS + if (self->db == NULL && rc == SQLITE_NOMEM) { + PyErr_NoMemory(); + return -1; + } if (rc != SQLITE_OK) { _pysqlite_seterror(state, self->db); return -1; From webhook-mailer at python.org Fri Oct 29 16:41:54 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:41:54 -0000 Subject: [Python-checkins] bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) (GH-29319) Message-ID: https://github.com/python/cpython/commit/8ea665c730cd86a321c558c012bef84f454efa4f commit: 8ea665c730cd86a321c558c012bef84f454efa4f branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T22:41:45+02:00 summary: bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) (GH-29319) (cherry picked from commit 3877fc02f7a8801ba5ce0e94b6075b3fdd9778d0) Co-authored-by: Ian Fisher files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index dc9aeb2ba3f59..16d2c2e85f42b 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1072,6 +1072,12 @@ If a timestamp stored in SQLite has a fractional part longer than 6 numbers, its value will be truncated to microsecond precision by the timestamp converter. +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. .. _sqlite3-controlling-transactions: From webhook-mailer at python.org Fri Oct 29 16:42:47 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:42:47 -0000 Subject: [Python-checkins] bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) (GH-29320) Message-ID: https://github.com/python/cpython/commit/e1560313d4d9bff8eba0f851ef325f7ee19f7ba9 commit: e1560313d4d9bff8eba0f851ef325f7ee19f7ba9 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T22:42:38+02:00 summary: bpo-45335: Add note to `sqlite3` docs about "timestamp" converter (GH-29200) (GH-29320) (cherry picked from commit 3877fc02f7a8801ba5ce0e94b6075b3fdd9778d0) Co-authored-by: Ian Fisher files: M Doc/library/sqlite3.rst diff --git a/Doc/library/sqlite3.rst b/Doc/library/sqlite3.rst index b9436daaebb4b..b24a2f0985489 100644 --- a/Doc/library/sqlite3.rst +++ b/Doc/library/sqlite3.rst @@ -1062,6 +1062,12 @@ If a timestamp stored in SQLite has a fractional part longer than 6 numbers, its value will be truncated to microsecond precision by the timestamp converter. +.. note:: + + The default "timestamp" converter ignores UTC offsets in the database and + always returns a naive :class:`datetime.datetime` object. To preserve UTC + offsets in timestamps, either leave converters disabled, or register an + offset-aware converter with :func:`register_converter`. .. _sqlite3-controlling-transactions: From webhook-mailer at python.org Fri Oct 29 16:45:42 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:45:42 -0000 Subject: [Python-checkins] bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) (GH-29322) Message-ID: https://github.com/python/cpython/commit/0538351fcaa320e62cb4ef92ec629b7e24a73e9c commit: 0538351fcaa320e62cb4ef92ec629b7e24a73e9c branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T22:45:33+02:00 summary: bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) (GH-29322) Co-authored-by: ?ukasz Langa (cherry picked from commit b17cfd150f4dc2816975d304a71110a2d445eaf0) Co-authored-by: andrei kulakov files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 237ca4b4b58ef..39d7e40dd915c 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -170,9 +170,10 @@ process and user. .. data:: environ - A :term:`mapping` object representing the string environment. For example, - ``environ['HOME']`` is the pathname of your home directory (on some platforms), - and is equivalent to ``getenv("HOME")`` in C. + A :term:`mapping` object where keys and values are strings that represent + the process environment. For example, ``environ['HOME']`` is the pathname + of your home directory (on some platforms), and is equivalent to + ``getenv("HOME")`` in C. This mapping is captured the first time the :mod:`os` module is imported, typically during Python startup as part of processing :file:`site.py`. Changes @@ -209,10 +210,10 @@ process and user. .. data:: environb - Bytes version of :data:`environ`: a :term:`mapping` object representing the - environment as byte strings. :data:`environ` and :data:`environb` are - synchronized (modify :data:`environb` updates :data:`environ`, and vice - versa). + Bytes version of :data:`environ`: a :term:`mapping` object where both keys + and values are :class:`bytes` objects representing the process environment. + :data:`environ` and :data:`environb` are synchronized (modifying + :data:`environb` updates :data:`environ`, and vice versa). :data:`environb` is only available if :data:`supports_bytes_environ` is ``True``. From webhook-mailer at python.org Fri Oct 29 16:53:10 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:53:10 -0000 Subject: [Python-checkins] bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) (GH-29321) Message-ID: https://github.com/python/cpython/commit/3ec1124de289496efabc43a02cc88b3c59e1e238 commit: 3ec1124de289496efabc43a02cc88b3c59e1e238 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: ambv date: 2021-10-29T22:52:59+02:00 summary: bpo-45600: Enhanced / clarified the docs for os.environ and os.environb (GH-29204) (GH-29321) Co-authored-by: ?ukasz Langa (cherry picked from commit b17cfd150f4dc2816975d304a71110a2d445eaf0) Co-authored-by: andrei kulakov files: M Doc/library/os.rst diff --git a/Doc/library/os.rst b/Doc/library/os.rst index 23e220c009d30..d4cc296fbf89a 100644 --- a/Doc/library/os.rst +++ b/Doc/library/os.rst @@ -102,9 +102,10 @@ process and user. .. data:: environ - A :term:`mapping` object representing the string environment. For example, - ``environ['HOME']`` is the pathname of your home directory (on some platforms), - and is equivalent to ``getenv("HOME")`` in C. + A :term:`mapping` object where keys and values are strings that represent + the process environment. For example, ``environ['HOME']`` is the pathname + of your home directory (on some platforms), and is equivalent to + ``getenv("HOME")`` in C. This mapping is captured the first time the :mod:`os` module is imported, typically during Python startup as part of processing :file:`site.py`. Changes @@ -141,10 +142,10 @@ process and user. .. data:: environb - Bytes version of :data:`environ`: a :term:`mapping` object representing the - environment as byte strings. :data:`environ` and :data:`environb` are - synchronized (modify :data:`environb` updates :data:`environ`, and vice - versa). + Bytes version of :data:`environ`: a :term:`mapping` object where both keys + and values are :class:`bytes` objects representing the process environment. + :data:`environ` and :data:`environb` are synchronized (modifying + :data:`environb` updates :data:`environ`, and vice versa). :data:`environb` is only available if :data:`supports_bytes_environ` is ``True``. From webhook-mailer at python.org Fri Oct 29 16:54:12 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:54:12 -0000 Subject: [Python-checkins] [3.10] bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) (GH-29323) Message-ID: https://github.com/python/cpython/commit/7e2c0a18b41cb906a354776e6ca52af81e39820f commit: 7e2c0a18b41cb906a354776e6ca52af81e39820f branch: 3.10 author: ?ukasz Langa committer: ambv date: 2021-10-29T22:54:07+02:00 summary: [3.10] bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) (GH-29323) (cherry picked from commit e2e62b3808691e15fa44b883270023e42dcad958) Co-authored-by: Erlend Egeberg Aasland files: A Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst M Modules/_sqlite/connection.c diff --git a/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst new file mode 100644 index 0000000000000..13a3b237434eb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst @@ -0,0 +1,2 @@ +:meth:`sqlite3.connect` now correctly raises :exc:`MemoryError` if the +underlying SQLite API signals memory error. Patch by Erlend E. Aasland. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 2cc5f53a30bc0..64610393ec1c3 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -113,6 +113,10 @@ pysqlite_connection_init(pysqlite_Connection *self, PyObject *args, Py_DECREF(database_obj); + if (self->db == NULL && rc == SQLITE_NOMEM) { + PyErr_NoMemory(); + return -1; + } if (rc != SQLITE_OK) { _pysqlite_seterror(self->db, NULL); return -1; From webhook-mailer at python.org Fri Oct 29 16:55:23 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 20:55:23 -0000 Subject: [Python-checkins] bpo-45379: clarify FROZEN_EXCLUDED and FROZEN_INVALID documentation (GH-29189) Message-ID: https://github.com/python/cpython/commit/c2d0ba722a7b3839685af968cf0c304a24cdf525 commit: c2d0ba722a7b3839685af968cf0c304a24cdf525 branch: main author: Filipe La?ns committer: ambv date: 2021-10-29T22:55:14+02:00 summary: bpo-45379: clarify FROZEN_EXCLUDED and FROZEN_INVALID documentation (GH-29189) Signed-off-by: Filipe La?ns Co-authored-by: ?ukasz Langa files: M Python/import.c diff --git a/Python/import.c b/Python/import.c index 48ea9129163c8..cdcb903c88207 100644 --- a/Python/import.c +++ b/Python/import.c @@ -1171,8 +1171,10 @@ typedef enum { FROZEN_BAD_NAME, // The given module name wasn't valid. FROZEN_NOT_FOUND, // It wasn't in PyImport_FrozenModules. FROZEN_DISABLED, // -X frozen_modules=off (and not essential) - FROZEN_EXCLUDED, // The PyImport_FrozenModules entry has NULL "code". - FROZEN_INVALID, // The PyImport_FrozenModules entry is bogus. + FROZEN_EXCLUDED, /* The PyImport_FrozenModules entry has NULL "code" + (module is present but marked as unimportable, stops search). */ + FROZEN_INVALID, /* The PyImport_FrozenModules entry is bogus + (eg. does not contain executable code). */ } frozen_status; static inline void @@ -1305,6 +1307,7 @@ find_frozen(PyObject *nameobj, struct frozen_info *info) return FROZEN_EXCLUDED; } if (p->code[0] == '\0' || p->size == 0) { + /* Does not contain executable code. */ return FROZEN_INVALID; } return FROZEN_OKAY; @@ -1315,6 +1318,7 @@ unmarshal_frozen_code(struct frozen_info *info) { PyObject *co = PyMarshal_ReadObjectFromString(info->data, info->size); if (co == NULL) { + /* Does not contain executable code. */ set_frozen_error(FROZEN_INVALID, info->nameobj); return NULL; } @@ -2214,6 +2218,7 @@ _imp_get_frozen_object_impl(PyObject *module, PyObject *name, info.nameobj = name; } if (info.size == 0) { + /* Does not contain executable code. */ set_frozen_error(FROZEN_INVALID, name); return NULL; } From webhook-mailer at python.org Fri Oct 29 17:02:30 2021 From: webhook-mailer at python.org (ambv) Date: Fri, 29 Oct 2021 21:02:30 -0000 Subject: [Python-checkins] [3.9] bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) (GH-29324) Message-ID: https://github.com/python/cpython/commit/ed807bf333cdc78b92c9861600acf1a435c52193 commit: ed807bf333cdc78b92c9861600acf1a435c52193 branch: 3.9 author: ?ukasz Langa committer: ambv date: 2021-10-29T23:02:19+02:00 summary: [3.9] bpo-45581: Raise `MemoryError` in `sqlite3.connect` if SQLite signals memory error (GH-29171) (GH-29324) (cherry picked from commit e2e62b3808691e15fa44b883270023e42dcad958) Co-authored-by: Erlend Egeberg Aasland files: A Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst M Modules/_sqlite/connection.c diff --git a/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst new file mode 100644 index 0000000000000..13a3b237434eb --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-22-21-57-02.bpo-45581.rlH6ay.rst @@ -0,0 +1,2 @@ +:meth:`sqlite3.connect` now correctly raises :exc:`MemoryError` if the +underlying SQLite API signals memory error. Patch by Erlend E. Aasland. diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index 68bf97389a7cd..30e333a4b86d8 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -134,6 +134,10 @@ int pysqlite_connection_init(pysqlite_Connection* self, PyObject* args, PyObject Py_DECREF(database_obj); + if (self->db == NULL && rc == SQLITE_NOMEM) { + PyErr_NoMemory(); + return -1; + } if (rc != SQLITE_OK) { _pysqlite_seterror(self->db, NULL); return -1; From webhook-mailer at python.org Fri Oct 29 18:08:23 2021 From: webhook-mailer at python.org (miss-islington) Date: Fri, 29 Oct 2021 22:08:23 -0000 Subject: [Python-checkins] bpo-10572: Move `sqlite3` tests to `Lib/test` (GH-29304) Message-ID: https://github.com/python/cpython/commit/62bf263a775f4444d8b5d5841cc09be3bd53e933 commit: 62bf263a775f4444d8b5d5841cc09be3bd53e933 branch: main author: Erlend Egeberg Aasland committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-29T15:08:19-07:00 summary: bpo-10572: Move `sqlite3` tests to `Lib/test` (GH-29304) Automerge-Triggered-By: GH:brettcannon files: A Lib/test/test_sqlite3/__init__.py A Lib/test/test_sqlite3/__main__.py A Lib/test/test_sqlite3/test_backup.py A Lib/test/test_sqlite3/test_dbapi.py A Lib/test/test_sqlite3/test_dump.py A Lib/test/test_sqlite3/test_factory.py A Lib/test/test_sqlite3/test_hooks.py A Lib/test/test_sqlite3/test_regression.py A Lib/test/test_sqlite3/test_transactions.py A Lib/test/test_sqlite3/test_types.py A Lib/test/test_sqlite3/test_userfunctions.py A Misc/NEWS.d/next/Library/2021-01-07-01-25-38.bpo-10572.gEEZ9z.rst D Lib/sqlite3/test/__init__.py D Lib/sqlite3/test/test_backup.py D Lib/sqlite3/test/test_dbapi.py D Lib/sqlite3/test/test_dump.py D Lib/sqlite3/test/test_factory.py D Lib/sqlite3/test/test_hooks.py D Lib/sqlite3/test/test_regression.py D Lib/sqlite3/test/test_transactions.py D Lib/sqlite3/test/test_types.py D Lib/sqlite3/test/test_userfunctions.py D Lib/test/test_sqlite.py M Makefile.pre.in M PCbuild/lib.pyproj diff --git a/Lib/sqlite3/test/__init__.py b/Lib/sqlite3/test/__init__.py deleted file mode 100644 index e69de29bb2d1d..0000000000000 diff --git a/Lib/test/test_sqlite.py b/Lib/test/test_sqlite.py deleted file mode 100644 index 9992a02e5d90a..0000000000000 --- a/Lib/test/test_sqlite.py +++ /dev/null @@ -1,20 +0,0 @@ -import test.support -from test.support import import_helper -from test.support import load_package_tests - -# Skip test if _sqlite3 module not installed -import_helper.import_module('_sqlite3') - -import unittest -import os -import sqlite3.test - -def load_tests(loader, tests, pattern): - if test.support.verbose: - print("test_sqlite: testing with version", - "{!r}, sqlite_version {!r}".format(sqlite3.version, - sqlite3.sqlite_version)) - return load_package_tests(os.path.dirname(sqlite3.test.__file__), loader, tests, pattern) - -if __name__ == "__main__": - unittest.main() diff --git a/Lib/test/test_sqlite3/__init__.py b/Lib/test/test_sqlite3/__init__.py new file mode 100644 index 0000000000000..099c01e3b3cc7 --- /dev/null +++ b/Lib/test/test_sqlite3/__init__.py @@ -0,0 +1,18 @@ +from test.support import import_helper, load_package_tests, verbose + +# Skip test if _sqlite3 module not installed. +import_helper.import_module('_sqlite3') + +import unittest +import os +import sqlite3 + +# Implement the unittest "load tests" protocol. +def load_tests(*args): + pkg_dir = os.path.dirname(__file__) + return load_package_tests(pkg_dir, *args) + +if verbose: + print("test_sqlite3: testing with version", + "{!r}, sqlite_version {!r}".format(sqlite3.version, + sqlite3.sqlite_version)) diff --git a/Lib/test/test_sqlite3/__main__.py b/Lib/test/test_sqlite3/__main__.py new file mode 100644 index 0000000000000..51eddc3c2fde0 --- /dev/null +++ b/Lib/test/test_sqlite3/__main__.py @@ -0,0 +1,5 @@ +from test.test_sqlite3 import load_tests # Needed for the "load tests" protocol. +import unittest + +if __name__ == "__main__": + unittest.main() diff --git a/Lib/sqlite3/test/test_backup.py b/Lib/test/test_sqlite3/test_backup.py similarity index 100% rename from Lib/sqlite3/test/test_backup.py rename to Lib/test/test_sqlite3/test_backup.py diff --git a/Lib/sqlite3/test/test_dbapi.py b/Lib/test/test_sqlite3/test_dbapi.py similarity index 100% rename from Lib/sqlite3/test/test_dbapi.py rename to Lib/test/test_sqlite3/test_dbapi.py diff --git a/Lib/sqlite3/test/test_dump.py b/Lib/test/test_sqlite3/test_dump.py similarity index 100% rename from Lib/sqlite3/test/test_dump.py rename to Lib/test/test_sqlite3/test_dump.py diff --git a/Lib/sqlite3/test/test_factory.py b/Lib/test/test_sqlite3/test_factory.py similarity index 100% rename from Lib/sqlite3/test/test_factory.py rename to Lib/test/test_sqlite3/test_factory.py diff --git a/Lib/sqlite3/test/test_hooks.py b/Lib/test/test_sqlite3/test_hooks.py similarity index 100% rename from Lib/sqlite3/test/test_hooks.py rename to Lib/test/test_sqlite3/test_hooks.py diff --git a/Lib/sqlite3/test/test_regression.py b/Lib/test/test_sqlite3/test_regression.py similarity index 100% rename from Lib/sqlite3/test/test_regression.py rename to Lib/test/test_sqlite3/test_regression.py diff --git a/Lib/sqlite3/test/test_transactions.py b/Lib/test/test_sqlite3/test_transactions.py similarity index 100% rename from Lib/sqlite3/test/test_transactions.py rename to Lib/test/test_sqlite3/test_transactions.py diff --git a/Lib/sqlite3/test/test_types.py b/Lib/test/test_sqlite3/test_types.py similarity index 100% rename from Lib/sqlite3/test/test_types.py rename to Lib/test/test_sqlite3/test_types.py diff --git a/Lib/sqlite3/test/test_userfunctions.py b/Lib/test/test_sqlite3/test_userfunctions.py similarity index 100% rename from Lib/sqlite3/test/test_userfunctions.py rename to Lib/test/test_sqlite3/test_userfunctions.py diff --git a/Makefile.pre.in b/Makefile.pre.in index 322800ae26e4c..7c3c046f711a5 100644 --- a/Makefile.pre.in +++ b/Makefile.pre.in @@ -1562,7 +1562,6 @@ TESTSUBDIRS= ctypes/test \ lib2to3/tests/data \ lib2to3/tests/data/fixers \ lib2to3/tests/data/fixers/myfixes \ - sqlite3/test \ test test/audiodata \ test/capath test/cjkencodings \ test/data test/decimaltestdata \ diff --git a/Misc/NEWS.d/next/Library/2021-01-07-01-25-38.bpo-10572.gEEZ9z.rst b/Misc/NEWS.d/next/Library/2021-01-07-01-25-38.bpo-10572.gEEZ9z.rst new file mode 100644 index 0000000000000..45f44bcbb110b --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-01-07-01-25-38.bpo-10572.gEEZ9z.rst @@ -0,0 +1,2 @@ +Move :mod:`sqlite3` tests to ``/Lib/test/test_sqlite3``. Patch by Erlend E. +Aasland. diff --git a/PCbuild/lib.pyproj b/PCbuild/lib.pyproj index 06e0a8b0d1c53..43c570f1dab37 100644 --- a/PCbuild/lib.pyproj +++ b/PCbuild/lib.pyproj @@ -1272,7 +1272,17 @@ - + + + + + + + + + + + From webhook-mailer at python.org Sat Oct 30 11:12:56 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Oct 2021 15:12:56 -0000 Subject: [Python-checkins] Fix docs source link for importlib.metadata (GH-29025) Message-ID: https://github.com/python/cpython/commit/aae18a17401dc36917c0f64f971d60ab1a5b477e commit: aae18a17401dc36917c0f64f971d60ab1a5b477e branch: main author: Philipp A committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-30T08:12:47-07:00 summary: Fix docs source link for importlib.metadata (GH-29025) The link broke for Python 3.10 since importlib.metadata was made from a module into a package I think this is trivial enough to not need a bpo issue. Automerge-Triggered-By: GH:jaraco files: M Doc/library/importlib.metadata.rst diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index 58c582d712413..99bcfeb2d1b55 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -11,7 +11,7 @@ .. versionchanged:: 3.10 ``importlib.metadata`` is no longer provisional. -**Source code:** :source:`Lib/importlib/metadata.py` +**Source code:** :source:`Lib/importlib/metadata/__init__.py` ``importlib.metadata`` is a library that provides for access to installed package metadata. Built in part on Python's import system, this library From webhook-mailer at python.org Sat Oct 30 11:32:23 2021 From: webhook-mailer at python.org (miss-islington) Date: Sat, 30 Oct 2021 15:32:23 -0000 Subject: [Python-checkins] Fix docs source link for importlib.metadata (GH-29025) Message-ID: https://github.com/python/cpython/commit/da222b7cc804adb1508f806d7fc6a2e65b8423e8 commit: da222b7cc804adb1508f806d7fc6a2e65b8423e8 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-30T08:32:14-07:00 summary: Fix docs source link for importlib.metadata (GH-29025) The link broke for Python 3.10 since importlib.metadata was made from a module into a package I think this is trivial enough to not need a bpo issue. Automerge-Triggered-By: GH:jaraco (cherry picked from commit aae18a17401dc36917c0f64f971d60ab1a5b477e) Co-authored-by: Philipp A files: M Doc/library/importlib.metadata.rst diff --git a/Doc/library/importlib.metadata.rst b/Doc/library/importlib.metadata.rst index 58c582d712413..99bcfeb2d1b55 100644 --- a/Doc/library/importlib.metadata.rst +++ b/Doc/library/importlib.metadata.rst @@ -11,7 +11,7 @@ .. versionchanged:: 3.10 ``importlib.metadata`` is no longer provisional. -**Source code:** :source:`Lib/importlib/metadata.py` +**Source code:** :source:`Lib/importlib/metadata/__init__.py` ``importlib.metadata`` is a library that provides for access to installed package metadata. Built in part on Python's import system, this library From webhook-mailer at python.org Sat Oct 30 22:14:03 2021 From: webhook-mailer at python.org (Fidget-Spinner) Date: Sun, 31 Oct 2021 02:14:03 -0000 Subject: [Python-checkins] bpo-45666: fixes warning with `swprintf` and `%s` (GH-29307) Message-ID: https://github.com/python/cpython/commit/aad48062ef8f983fbb95f9dc0c3c3cef9c89df02 commit: aad48062ef8f983fbb95f9dc0c3c3cef9c89df02 branch: main author: Nikita Sobolev committer: Fidget-Spinner <28750310+Fidget-Spinner at users.noreply.github.com> date: 2021-10-31T10:13:54+08:00 summary: bpo-45666: fixes warning with `swprintf` and `%s` (GH-29307) files: A Misc/NEWS.d/next/Build/2021-10-29-12-54-53.bpo-45666.w2G63u.rst M Programs/_testembed.c diff --git a/Misc/NEWS.d/next/Build/2021-10-29-12-54-53.bpo-45666.w2G63u.rst b/Misc/NEWS.d/next/Build/2021-10-29-12-54-53.bpo-45666.w2G63u.rst new file mode 100644 index 0000000000000..8a3e6feded856 --- /dev/null +++ b/Misc/NEWS.d/next/Build/2021-10-29-12-54-53.bpo-45666.w2G63u.rst @@ -0,0 +1 @@ +Fix warning of ``swprintf`` and ``%s`` usage in ``_testembed.c`` diff --git a/Programs/_testembed.c b/Programs/_testembed.c index 6fe18d93a73ae..1ed3bd00edff6 100644 --- a/Programs/_testembed.c +++ b/Programs/_testembed.c @@ -1733,7 +1733,7 @@ static int check_use_frozen_modules(const char *rawval) if (rawval == NULL) { wcscpy(optval, L"frozen_modules"); } - else if (swprintf(optval, 100, L"frozen_modules=%s", rawval) < 0) { + else if (swprintf(optval, 100, L"frozen_modules=%S", rawval) < 0) { error("rawval is too long"); return -1; } From webhook-mailer at python.org Sun Oct 31 04:22:21 2021 From: webhook-mailer at python.org (serhiy-storchaka) Date: Sun, 31 Oct 2021 08:22:21 -0000 Subject: [Python-checkins] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) Message-ID: https://github.com/python/cpython/commit/634984d7dbdd91e0a51a793eed4d870e139ae1e0 commit: 634984d7dbdd91e0a51a793eed4d870e139ae1e0 branch: main author: Serhiy Storchaka committer: serhiy-storchaka date: 2021-10-31T10:22:16+02:00 summary: bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) Literal[True, 2] is no longer equal to Literal[1, 2]. files: A Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index b1414dc82bae0..90d6bea59f899 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -714,6 +714,8 @@ def test_equal(self): self.assertNotEqual(Literal[True], Literal[1]) self.assertNotEqual(Literal[1], Literal[2]) self.assertNotEqual(Literal[1, True], Literal[1]) + self.assertNotEqual(Literal[1, True], Literal[1, 1]) + self.assertNotEqual(Literal[1, 2], Literal[True, 2]) self.assertEqual(Literal[1], Literal[1]) self.assertEqual(Literal[1, 2], Literal[2, 1]) self.assertEqual(Literal[1, 2, 3], Literal[1, 2, 3, 3]) @@ -4963,6 +4965,8 @@ def test_special_attrs(self): typing.Concatenate[Any, SpecialAttrsP]: 'Concatenate', typing.Final[Any]: 'Final', typing.Literal[Any]: 'Literal', + typing.Literal[1, 2]: 'Literal', + typing.Literal[True, 2]: 'Literal', typing.Optional[Any]: 'Optional', typing.TypeGuard[Any]: 'TypeGuard', typing.Union[Any]: 'Any', diff --git a/Lib/typing.py b/Lib/typing.py index 78d973d2bba05..031aa24eaf442 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -411,9 +411,10 @@ def __getitem__(self, parameters): class _LiteralSpecialForm(_SpecialForm, _root=True): - @_tp_cache(typed=True) def __getitem__(self, parameters): - return self._getitem(self, parameters) + if not isinstance(parameters, tuple): + parameters = (parameters,) + return self._getitem(self, *parameters) @_SpecialForm @@ -536,7 +537,8 @@ def Optional(self, parameters): return Union[arg, type(None)] @_LiteralSpecialForm -def Literal(self, parameters): + at _tp_cache(typed=True) +def Literal(self, *parameters): """Special typing form to define literal types (a.k.a. value types). This form can be used to indicate to type checkers that the corresponding @@ -559,9 +561,6 @@ def open_helper(file: str, mode: MODE) -> str: """ # There is no '_type_check' call because arguments to Literal[...] are # values, not types. - if not isinstance(parameters, tuple): - parameters = (parameters,) - parameters = _flatten_literal_params(parameters) try: diff --git a/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst b/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst new file mode 100644 index 0000000000000..a644492a12d17 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst @@ -0,0 +1,2 @@ +Fix caching of multi-value :data:`typing.Literal`. ``Literal[True, 2]`` is no +longer equal to ``Literal[1, 2]``. From webhook-mailer at python.org Sun Oct 31 04:43:45 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 08:43:45 -0000 Subject: [Python-checkins] bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) Message-ID: https://github.com/python/cpython/commit/3997f3ce8ab15269fc800062f75411865dbc0d55 commit: 3997f3ce8ab15269fc800062f75411865dbc0d55 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T01:43:40-07:00 summary: bpo-45679: Fix caching of multi-value typing.Literal (GH-29334) Literal[True, 2] is no longer equal to Literal[1, 2]. (cherry picked from commit 634984d7dbdd91e0a51a793eed4d870e139ae1e0) Co-authored-by: Serhiy Storchaka files: A Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst M Lib/test/test_typing.py M Lib/typing.py diff --git a/Lib/test/test_typing.py b/Lib/test/test_typing.py index 762c0cc8f6be0..fdec29ea58773 100644 --- a/Lib/test/test_typing.py +++ b/Lib/test/test_typing.py @@ -708,6 +708,8 @@ def test_equal(self): self.assertNotEqual(Literal[True], Literal[1]) self.assertNotEqual(Literal[1], Literal[2]) self.assertNotEqual(Literal[1, True], Literal[1]) + self.assertNotEqual(Literal[1, True], Literal[1, 1]) + self.assertNotEqual(Literal[1, 2], Literal[True, 2]) self.assertEqual(Literal[1], Literal[1]) self.assertEqual(Literal[1, 2], Literal[2, 1]) self.assertEqual(Literal[1, 2, 3], Literal[1, 2, 3, 3]) @@ -4917,6 +4919,8 @@ def test_special_attrs(self): typing.Concatenate[Any, SpecialAttrsP]: 'Concatenate', typing.Final[Any]: 'Final', typing.Literal[Any]: 'Literal', + typing.Literal[1, 2]: 'Literal', + typing.Literal[True, 2]: 'Literal', typing.Optional[Any]: 'Optional', typing.TypeGuard[Any]: 'TypeGuard', typing.Union[Any]: 'Any', diff --git a/Lib/typing.py b/Lib/typing.py index f842fc23da6c6..21caabb5d3e67 100644 --- a/Lib/typing.py +++ b/Lib/typing.py @@ -403,9 +403,10 @@ def __getitem__(self, parameters): class _LiteralSpecialForm(_SpecialForm, _root=True): - @_tp_cache(typed=True) def __getitem__(self, parameters): - return self._getitem(self, parameters) + if not isinstance(parameters, tuple): + parameters = (parameters,) + return self._getitem(self, *parameters) @_SpecialForm @@ -528,7 +529,8 @@ def Optional(self, parameters): return Union[arg, type(None)] @_LiteralSpecialForm -def Literal(self, parameters): + at _tp_cache(typed=True) +def Literal(self, *parameters): """Special typing form to define literal types (a.k.a. value types). This form can be used to indicate to type checkers that the corresponding @@ -551,9 +553,6 @@ def open_helper(file: str, mode: MODE) -> str: """ # There is no '_type_check' call because arguments to Literal[...] are # values, not types. - if not isinstance(parameters, tuple): - parameters = (parameters,) - parameters = _flatten_literal_params(parameters) try: diff --git a/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst b/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst new file mode 100644 index 0000000000000..a644492a12d17 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-10-30-21-11-37.bpo-45679.Dq8Cpu.rst @@ -0,0 +1,2 @@ +Fix caching of multi-value :data:`typing.Literal`. ``Literal[True, 2]`` is no +longer equal to ``Literal[1, 2]``. From webhook-mailer at python.org Sun Oct 31 06:01:33 2021 From: webhook-mailer at python.org (corona10) Date: Sun, 31 Oct 2021 10:01:33 -0000 Subject: [Python-checkins] bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251) Message-ID: https://github.com/python/cpython/commit/39b4d5938ce781af41f8c9da72dee46095a78642 commit: 39b4d5938ce781af41f8c9da72dee46095a78642 branch: main author: Erlend Egeberg Aasland committer: corona10 date: 2021-10-31T19:01:19+09:00 summary: bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251) files: M Modules/_sqlite/module.c diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index e41ac0fc1ab73..912851ba775dc 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -348,49 +348,56 @@ pysqlite_error_name(int rc) return NULL; } -static int add_integer_constants(PyObject *module) { - int ret = 0; - - ret += PyModule_AddIntMacro(module, PARSE_DECLTYPES); - ret += PyModule_AddIntMacro(module, PARSE_COLNAMES); - ret += PyModule_AddIntMacro(module, SQLITE_DENY); - ret += PyModule_AddIntMacro(module, SQLITE_IGNORE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_DELETE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_INSERT); - ret += PyModule_AddIntMacro(module, SQLITE_PRAGMA); - ret += PyModule_AddIntMacro(module, SQLITE_READ); - ret += PyModule_AddIntMacro(module, SQLITE_SELECT); - ret += PyModule_AddIntMacro(module, SQLITE_TRANSACTION); - ret += PyModule_AddIntMacro(module, SQLITE_UPDATE); - ret += PyModule_AddIntMacro(module, SQLITE_ATTACH); - ret += PyModule_AddIntMacro(module, SQLITE_DETACH); - ret += PyModule_AddIntMacro(module, SQLITE_ALTER_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_REINDEX); - ret += PyModule_AddIntMacro(module, SQLITE_ANALYZE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_VTABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_VTABLE); - ret += PyModule_AddIntMacro(module, SQLITE_FUNCTION); - ret += PyModule_AddIntMacro(module, SQLITE_SAVEPOINT); +static int +add_integer_constants(PyObject *module) { +#define ADD_INT(ival) \ + do { \ + if (PyModule_AddIntConstant(module, #ival, ival) < 0) { \ + return -1; \ + } \ + } while (0); \ + + ADD_INT(PARSE_DECLTYPES); + ADD_INT(PARSE_COLNAMES); + ADD_INT(SQLITE_DENY); + ADD_INT(SQLITE_IGNORE); + ADD_INT(SQLITE_CREATE_INDEX); + ADD_INT(SQLITE_CREATE_TABLE); + ADD_INT(SQLITE_CREATE_TEMP_INDEX); + ADD_INT(SQLITE_CREATE_TEMP_TABLE); + ADD_INT(SQLITE_CREATE_TEMP_TRIGGER); + ADD_INT(SQLITE_CREATE_TEMP_VIEW); + ADD_INT(SQLITE_CREATE_TRIGGER); + ADD_INT(SQLITE_CREATE_VIEW); + ADD_INT(SQLITE_DELETE); + ADD_INT(SQLITE_DROP_INDEX); + ADD_INT(SQLITE_DROP_TABLE); + ADD_INT(SQLITE_DROP_TEMP_INDEX); + ADD_INT(SQLITE_DROP_TEMP_TABLE); + ADD_INT(SQLITE_DROP_TEMP_TRIGGER); + ADD_INT(SQLITE_DROP_TEMP_VIEW); + ADD_INT(SQLITE_DROP_TRIGGER); + ADD_INT(SQLITE_DROP_VIEW); + ADD_INT(SQLITE_INSERT); + ADD_INT(SQLITE_PRAGMA); + ADD_INT(SQLITE_READ); + ADD_INT(SQLITE_SELECT); + ADD_INT(SQLITE_TRANSACTION); + ADD_INT(SQLITE_UPDATE); + ADD_INT(SQLITE_ATTACH); + ADD_INT(SQLITE_DETACH); + ADD_INT(SQLITE_ALTER_TABLE); + ADD_INT(SQLITE_REINDEX); + ADD_INT(SQLITE_ANALYZE); + ADD_INT(SQLITE_CREATE_VTABLE); + ADD_INT(SQLITE_DROP_VTABLE); + ADD_INT(SQLITE_FUNCTION); + ADD_INT(SQLITE_SAVEPOINT); #if SQLITE_VERSION_NUMBER >= 3008003 - ret += PyModule_AddIntMacro(module, SQLITE_RECURSIVE); + ADD_INT(SQLITE_RECURSIVE); #endif - return ret; +#undef ADD_INT + return 0; } struct PyModuleDef _sqlite3module = { From webhook-mailer at python.org Sun Oct 31 07:22:30 2021 From: webhook-mailer at python.org (corona10) Date: Sun, 31 Oct 2021 11:22:30 -0000 Subject: [Python-checkins] [3.10] bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251). (GH-29343) Message-ID: https://github.com/python/cpython/commit/ed91f959b032951620f32bcd93ff27223699a86a commit: ed91f959b032951620f32bcd93ff27223699a86a branch: 3.10 author: Erlend Egeberg Aasland committer: corona10 date: 2021-10-31T20:22:22+09:00 summary: [3.10] bpo-45634: Don't combine error checks when adding `sqlite3` int constants (GH-29251). (GH-29343) files: M Modules/_sqlite/module.c diff --git a/Modules/_sqlite/module.c b/Modules/_sqlite/module.c index a5e5525481f75..ba70a6c51f440 100644 --- a/Modules/_sqlite/module.c +++ b/Modules/_sqlite/module.c @@ -273,51 +273,58 @@ static PyMethodDef module_methods[] = { {NULL, NULL} }; -static int add_integer_constants(PyObject *module) { - int ret = 0; - - ret += PyModule_AddIntMacro(module, PARSE_DECLTYPES); - ret += PyModule_AddIntMacro(module, PARSE_COLNAMES); - ret += PyModule_AddIntMacro(module, SQLITE_OK); - ret += PyModule_AddIntMacro(module, SQLITE_DENY); - ret += PyModule_AddIntMacro(module, SQLITE_IGNORE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TEMP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_DELETE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_INDEX); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TEMP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_TRIGGER); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_VIEW); - ret += PyModule_AddIntMacro(module, SQLITE_INSERT); - ret += PyModule_AddIntMacro(module, SQLITE_PRAGMA); - ret += PyModule_AddIntMacro(module, SQLITE_READ); - ret += PyModule_AddIntMacro(module, SQLITE_SELECT); - ret += PyModule_AddIntMacro(module, SQLITE_TRANSACTION); - ret += PyModule_AddIntMacro(module, SQLITE_UPDATE); - ret += PyModule_AddIntMacro(module, SQLITE_ATTACH); - ret += PyModule_AddIntMacro(module, SQLITE_DETACH); - ret += PyModule_AddIntMacro(module, SQLITE_ALTER_TABLE); - ret += PyModule_AddIntMacro(module, SQLITE_REINDEX); - ret += PyModule_AddIntMacro(module, SQLITE_ANALYZE); - ret += PyModule_AddIntMacro(module, SQLITE_CREATE_VTABLE); - ret += PyModule_AddIntMacro(module, SQLITE_DROP_VTABLE); - ret += PyModule_AddIntMacro(module, SQLITE_FUNCTION); - ret += PyModule_AddIntMacro(module, SQLITE_SAVEPOINT); +static int +add_integer_constants(PyObject *module) { +#define ADD_INT(ival) \ + do { \ + if (PyModule_AddIntConstant(module, #ival, ival) < 0) { \ + return -1; \ + } \ + } while (0); \ + + ADD_INT(PARSE_DECLTYPES); + ADD_INT(PARSE_COLNAMES); + ADD_INT(SQLITE_OK); + ADD_INT(SQLITE_DENY); + ADD_INT(SQLITE_IGNORE); + ADD_INT(SQLITE_CREATE_INDEX); + ADD_INT(SQLITE_CREATE_TABLE); + ADD_INT(SQLITE_CREATE_TEMP_INDEX); + ADD_INT(SQLITE_CREATE_TEMP_TABLE); + ADD_INT(SQLITE_CREATE_TEMP_TRIGGER); + ADD_INT(SQLITE_CREATE_TEMP_VIEW); + ADD_INT(SQLITE_CREATE_TRIGGER); + ADD_INT(SQLITE_CREATE_VIEW); + ADD_INT(SQLITE_DELETE); + ADD_INT(SQLITE_DROP_INDEX); + ADD_INT(SQLITE_DROP_TABLE); + ADD_INT(SQLITE_DROP_TEMP_INDEX); + ADD_INT(SQLITE_DROP_TEMP_TABLE); + ADD_INT(SQLITE_DROP_TEMP_TRIGGER); + ADD_INT(SQLITE_DROP_TEMP_VIEW); + ADD_INT(SQLITE_DROP_TRIGGER); + ADD_INT(SQLITE_DROP_VIEW); + ADD_INT(SQLITE_INSERT); + ADD_INT(SQLITE_PRAGMA); + ADD_INT(SQLITE_READ); + ADD_INT(SQLITE_SELECT); + ADD_INT(SQLITE_TRANSACTION); + ADD_INT(SQLITE_UPDATE); + ADD_INT(SQLITE_ATTACH); + ADD_INT(SQLITE_DETACH); + ADD_INT(SQLITE_ALTER_TABLE); + ADD_INT(SQLITE_REINDEX); + ADD_INT(SQLITE_ANALYZE); + ADD_INT(SQLITE_CREATE_VTABLE); + ADD_INT(SQLITE_DROP_VTABLE); + ADD_INT(SQLITE_FUNCTION); + ADD_INT(SQLITE_SAVEPOINT); #if SQLITE_VERSION_NUMBER >= 3008003 - ret += PyModule_AddIntMacro(module, SQLITE_RECURSIVE); + ADD_INT(SQLITE_RECURSIVE); #endif - ret += PyModule_AddIntMacro(module, SQLITE_DONE); - return ret; + ADD_INT(SQLITE_DONE); +#undef ADD_INT + return 0; } static struct PyModuleDef _sqlite3module = { From webhook-mailer at python.org Sun Oct 31 10:57:29 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 14:57:29 -0000 Subject: [Python-checkins] bpo-45516: fix Traversable.name documentation (GH-29194) Message-ID: https://github.com/python/cpython/commit/bc8fd7c9c24eb71217ebdb8a12cf38e9dc4215b2 commit: bc8fd7c9c24eb71217ebdb8a12cf38e9dc4215b2 branch: main author: Filipe La?ns committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T07:57:21-07:00 summary: bpo-45516: fix Traversable.name documentation (GH-29194) Here we go again... Sorrry! Signed-off-by: Filipe La?ns Automerge-Triggered-By: GH:jaraco files: M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 7a656fcb091da..5b4fba381ce5b 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -815,9 +815,9 @@ ABC hierarchy:: .. versionadded:: 3.9 - .. abstractmethod:: name() + .. attribute:: name - The base name of this object without any parent references. + Abstract. The base name of this object without any parent references. .. abstractmethod:: iterdir() From webhook-mailer at python.org Sun Oct 31 13:44:31 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 17:44:31 -0000 Subject: [Python-checkins] Move field width to the right field (GH-29344) Message-ID: https://github.com/python/cpython/commit/ee464322eb5ede56db6b501279f700638c5c9103 commit: ee464322eb5ede56db6b501279f700638c5c9103 branch: main author: Olaf van der Spek committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T10:44:23-07:00 summary: Move field width to the right field (GH-29344) Automerge-Triggered-By: GH:vsajip files: M Doc/library/logging.rst diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index ee0276aadcf3d..3f3a8534f54d0 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -203,7 +203,7 @@ is the module's name in the Python package namespace. attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logger = logging.getLogger('tcpserver') @@ -1000,7 +1000,7 @@ functions. be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logging.warning('Protocol problem: %s', 'connection reset', extra=d) From webhook-mailer at python.org Sun Oct 31 15:26:58 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 19:26:58 -0000 Subject: [Python-checkins] bpo-45516: add protocol description to the Traversable documentation (GH-29039) Message-ID: https://github.com/python/cpython/commit/89b1237ffbd2df2721dc4959e3ccb1106f3555b5 commit: 89b1237ffbd2df2721dc4959e3ccb1106f3555b5 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T12:26:49-07:00 summary: bpo-45516: add protocol description to the Traversable documentation (GH-29039) * bpo-45516: add protocol description to the Traversable documentation Signed-off-by: Filipe La?ns * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs * Update Lib/importlib/abc.py * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs (cherry picked from commit 4d03de3329ed8daa9c1107b1aedbb0fa280bddb6) Co-authored-by: Filipe La?ns files: A Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index b5ee7a6b9659a..3576941efa46d 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -815,6 +815,46 @@ ABC hierarchy:: .. versionadded:: 3.9 + .. abstractmethod:: name() + + The base name of this object without any parent references. + + .. abstractmethod:: iterdir() + + Yield Traversable objects in self. + + .. abstractmethod:: is_dir() + + Return True if self is a directory. + + .. abstractmethod:: is_file() + + Return True if self is a file. + + .. abstractmethod:: joinpath(child) + + Return Traversable child in self. + + .. abstractmethod:: __truediv__(child) + + Return Traversable child in self. + + .. abstractmethod:: open(mode='r', *args, **kwargs) + + *mode* may be 'r' or 'rb' to open as text or binary. Return a handle + suitable for reading (same as :attr:`pathlib.Path.open`). + + When opening as text, accepts encoding parameters such as those + accepted by :attr:`io.TextIOWrapper`. + + .. method:: read_bytes() + + Read contents of self as bytes. + + .. method:: read_text(encoding=None) + + Read contents of self as text. + .. class:: TraversableResources diff --git a/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst new file mode 100644 index 0000000000000..98f5d3432db05 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst @@ -0,0 +1,2 @@ +Add protocol description to the :class:`importlib.abc.Traversable` +documentation. From webhook-mailer at python.org Sun Oct 31 15:27:59 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 19:27:59 -0000 Subject: [Python-checkins] bpo-45516: add protocol description to the Traversable documentation (GH-29039) Message-ID: https://github.com/python/cpython/commit/b04b307e0ebad0d9c7cc311fe6018b8d31b2ac8d commit: b04b307e0ebad0d9c7cc311fe6018b8d31b2ac8d branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T12:27:49-07:00 summary: bpo-45516: add protocol description to the Traversable documentation (GH-29039) * bpo-45516: add protocol description to the Traversable documentation Signed-off-by: Filipe La?ns * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs * Update Lib/importlib/abc.py * Update Doc/library/importlib.rst Co-authored-by: Jason R. Coombs Co-authored-by: Jason R. Coombs (cherry picked from commit 4d03de3329ed8daa9c1107b1aedbb0fa280bddb6) Co-authored-by: Filipe La?ns files: A Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst M Doc/library/importlib.rst diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst index 69e311298221c..c7fbcb21274a1 100644 --- a/Doc/library/importlib.rst +++ b/Doc/library/importlib.rst @@ -805,6 +805,46 @@ ABC hierarchy:: .. versionadded:: 3.9 + .. abstractmethod:: name() + + The base name of this object without any parent references. + + .. abstractmethod:: iterdir() + + Yield Traversable objects in self. + + .. abstractmethod:: is_dir() + + Return True if self is a directory. + + .. abstractmethod:: is_file() + + Return True if self is a file. + + .. abstractmethod:: joinpath(child) + + Return Traversable child in self. + + .. abstractmethod:: __truediv__(child) + + Return Traversable child in self. + + .. abstractmethod:: open(mode='r', *args, **kwargs) + + *mode* may be 'r' or 'rb' to open as text or binary. Return a handle + suitable for reading (same as :attr:`pathlib.Path.open`). + + When opening as text, accepts encoding parameters such as those + accepted by :attr:`io.TextIOWrapper`. + + .. method:: read_bytes() + + Read contents of self as bytes. + + .. method:: read_text(encoding=None) + + Read contents of self as text. + .. class:: TraversableResources diff --git a/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst new file mode 100644 index 0000000000000..98f5d3432db05 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-18-20-12-18.bpo-45516.EJh4K8.rst @@ -0,0 +1,2 @@ +Add protocol description to the :class:`importlib.abc.Traversable` +documentation. From webhook-mailer at python.org Sun Oct 31 15:53:21 2021 From: webhook-mailer at python.org (Mariatta) Date: Sun, 31 Oct 2021 19:53:21 -0000 Subject: [Python-checkins] Updated README with Python 3.11 release info (GH-29338) Message-ID: https://github.com/python/cpython/commit/d3870eaa9770148495703882ca15e88465219a53 commit: d3870eaa9770148495703882ca15e88465219a53 branch: main author: Jacob Walls committer: Mariatta date: 2021-10-31T12:53:08-07:00 summary: Updated README with Python 3.11 release info (GH-29338) files: M README.rst diff --git a/README.rst b/README.rst index c2b26a4651511..a789408d78d53 100644 --- a/README.rst +++ b/README.rst @@ -241,7 +241,7 @@ All current PEPs, as well as guidelines for submitting a new PEP, are listed at Release Schedule ---------------- -See :pep:`619` for Python 3.10 release details. +See :pep:`664` for Python 3.11 release details. Copyright and License Information From webhook-mailer at python.org Sun Oct 31 16:25:30 2021 From: webhook-mailer at python.org (vsajip) Date: Sun, 31 Oct 2021 20:25:30 -0000 Subject: [Python-checkins] [3.9] Move field width to the right field (GH-29344) (GH-29350) Message-ID: https://github.com/python/cpython/commit/da04bef5f7bcbce48e7e1b8c268b8899a0a42140 commit: da04bef5f7bcbce48e7e1b8c268b8899a0a42140 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vsajip date: 2021-10-31T20:25:21Z summary: [3.9] Move field width to the right field (GH-29344) (GH-29350) Co-authored-by: Olaf van der Spek files: M Doc/library/logging.rst diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 1a4c84e4a3a21..e72a898a11363 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -203,7 +203,7 @@ is the module's name in the Python package namespace. attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logger = logging.getLogger('tcpserver') @@ -993,7 +993,7 @@ functions. be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logging.warning('Protocol problem: %s', 'connection reset', extra=d) From webhook-mailer at python.org Sun Oct 31 16:25:52 2021 From: webhook-mailer at python.org (vsajip) Date: Sun, 31 Oct 2021 20:25:52 -0000 Subject: [Python-checkins] [3.10] Move field width to the right field (GH-29344) (GH-29351) Message-ID: https://github.com/python/cpython/commit/ed34965d66b11c3bae43b20671aa796299a56451 commit: ed34965d66b11c3bae43b20671aa796299a56451 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington at users.noreply.github.com> committer: vsajip date: 2021-10-31T20:25:44Z summary: [3.10] Move field width to the right field (GH-29344) (GH-29351) Co-authored-by: Olaf van der Spek files: M Doc/library/logging.rst diff --git a/Doc/library/logging.rst b/Doc/library/logging.rst index 9de115a92c293..bb1bbf0e3708c 100644 --- a/Doc/library/logging.rst +++ b/Doc/library/logging.rst @@ -203,7 +203,7 @@ is the module's name in the Python package namespace. attributes can then be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logger = logging.getLogger('tcpserver') @@ -1000,7 +1000,7 @@ functions. be used as you like. For example, they could be incorporated into logged messages. For example:: - FORMAT = '%(asctime)-15s %(clientip)s %(user)-8s %(message)s' + FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s' logging.basicConfig(format=FORMAT) d = {'clientip': '192.168.0.1', 'user': 'fbloggs'} logging.warning('Protocol problem: %s', 'connection reset', extra=d) From webhook-mailer at python.org Sun Oct 31 16:51:43 2021 From: webhook-mailer at python.org (miss-islington) Date: Sun, 31 Oct 2021 20:51:43 -0000 Subject: [Python-checkins] bpo-45618: Update Sphinx version used to build the documentation to 4.2.0 (GH-29256) Message-ID: https://github.com/python/cpython/commit/14a4fce457033412278ca9a056787db424310dc3 commit: 14a4fce457033412278ca9a056787db424310dc3 branch: main author: m-aciek committer: miss-islington <31488909+miss-islington at users.noreply.github.com> date: 2021-10-31T13:51:33-07:00 summary: bpo-45618: Update Sphinx version used to build the documentation to 4.2.0 (GH-29256) Updating version of Sphinx to most recent one, 4.2.0. Documentation builds without warnings. Sphinx 4.2.0 release notes: https://www.sphinx-doc.org/en/master/changes.html#release-4-2-0-released-sep-12-2021. cc @JulienPalard Automerge-Triggered-By: GH:JulienPalard files: A Misc/NEWS.d/next/Documentation/2021-10-31-20-35-06.bpo-45618.RTcNXF.rst M Doc/requirements.txt diff --git a/Doc/requirements.txt b/Doc/requirements.txt index dd3c8e62237cf..785da2c321784 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -3,11 +3,7 @@ # Sphinx version is pinned so that new versions that introduce new warnings # won't suddenly cause build failures. Updating the version is fine as long # as no warnings are raised by doing so. -sphinx==3.2.1 -# Docutils version is pinned to a version compatible with Sphinx -# version 3.2.1. It can be removed after bumping Sphinx version to at -# least 3.5.4. -docutils==0.17.1 +sphinx==4.2.0 blurb diff --git a/Misc/NEWS.d/next/Documentation/2021-10-31-20-35-06.bpo-45618.RTcNXF.rst b/Misc/NEWS.d/next/Documentation/2021-10-31-20-35-06.bpo-45618.RTcNXF.rst new file mode 100644 index 0000000000000..47242b9382809 --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2021-10-31-20-35-06.bpo-45618.RTcNXF.rst @@ -0,0 +1,2 @@ +Update Sphinx version used to build the documentation to 4.2.0. +Patch by Maciej Olko. \ No newline at end of file