Mailman 3 March 2022 - Python-checkins

bpo-42340: Document issues around KeyboardInterrupt (GH-23255)
by miss-islington 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/c26af2bc531eb114c378cdad81935f6e83… commit: c26af2bc531eb114c378cdad81935f6e838a7ee0 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2022-03-29T14:48:10-07:00 summary: bpo-42340: Document issues around KeyboardInterrupt (GH-23255) Update documentation to note that in some circumstances, KeyboardInterrupt may cause code to enter an inconsistent state. Also document sample workaround to avoid KeyboardInterrupt, if needed. (cherry picked from commit d0906c90fcfbc4cfb9bb963eaa6bb152dd543b56) Co-authored-by: benfogle <benfogle(a)gmail.com> files: A Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst M Doc/library/exceptions.rst M Doc/library/signal.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index c4945679476a4..688eeb31b2ab0 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -234,6 +234,15 @@ The following exceptions are the exceptions that are usually raised. accidentally caught by code that catches :exc:`Exception` and thus prevent the interpreter from exiting. + .. note:: + + Catching a :exc:`KeyboardInterrupt` requires special consideration. + Because it can be raised at unpredictable points, it may, in some + circumstances, leave the running program in an inconsistent state. It is + generally best to allow :exc:`KeyboardInterrupt` to end the program as + quickly as possible or avoid raising it entirely. (See + :ref:`handlers-and-exceptions`.) + .. exception:: MemoryError diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index e1daeff48ad8f..faea9d42a3fbd 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -46,6 +46,9 @@ This has consequences: arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes. +* If the handler raises an exception, it will be raised "out of thin air" in + the main thread. See the :ref:`note below <handlers-and-exceptions>` for a + discussion. .. _signals-and-threads: @@ -677,3 +680,70 @@ Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in order to avoid :exc:`BrokenPipeError`. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. + +.. _handlers-and-exceptions: + +Note on Signal Handlers and Exceptions +-------------------------------------- + +If a signal handler raises an exception, the exception will be propagated to +the main thread and may be raised after any :term:`bytecode` instruction. Most +notably, a :exc:`KeyboardInterrupt` may appear at any point during execution. +Most Python code, including the standard library, cannot be made robust against +this, and so a :exc:`KeyboardInterrupt` (or any other exception resulting from +a signal handler) may on rare occasions put the program in an unexpected state. + +To illustrate this issue, consider the following code:: + + class SpamContext: + def __init__(self): + self.lock = threading.Lock() + + def __enter__(self): + # If KeyboardInterrupt occurs here, everything is fine + self.lock.acquire() + # If KeyboardInterrupt occcurs here, __exit__ will not be called + ... + # KeyboardInterrupt could occur just before the function returns + + def __exit__(self, exc_type, exc_val, exc_tb): + ... + self.lock.release() + +For many programs, especially those that merely want to exit on +:exc:`KeyboardInterrupt`, this is not a problem, but applications that are +complex or require high reliability should avoid raising exceptions from signal +handlers. They should also avoid catching :exc:`KeyboardInterrupt` as a means +of gracefully shutting down. Instead, they should install their own +:const:`SIGINT` handler. Below is an example of an HTTP server that avoids +:exc:`KeyboardInterrupt`:: + + import signal + import socket + from selectors import DefaultSelector, EVENT_READ + from http.server import HTTPServer, SimpleHTTPRequestHandler + + interrupt_read, interrupt_write = socket.socketpair() + + def handler(signum, frame): + print('Signal handler called with signal', signum) + interrupt_write.send(b'\0') + signal.signal(signal.SIGINT, handler) + + def serve_forever(httpd): + sel = DefaultSelector() + sel.register(interrupt_read, EVENT_READ) + sel.register(httpd, EVENT_READ) + + while True: + for key, _ in sel.select(): + if key.fileobj == interrupt_read: + interrupt_read.recv(1) + return + if key.fileobj == httpd: + httpd.handle_request() + + print("Serving on port 8000") + httpd = HTTPServer(('', 8000), SimpleHTTPRequestHandler) + serve_forever(httpd) + print("Shutdown...") diff --git a/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst new file mode 100644 index 0000000000000..aa6857497383c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst @@ -0,0 +1,3 @@ +Document that in some circumstances :exc:`KeyboardInterrupt` may cause the +code to enter an inconsistent state. Provided a sample workaround to avoid +it if needed.

1 0

bpo-42340: Document issues around KeyboardInterrupt (GH-23255)
by miss-islington 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/66cde7c51a871a86cf8667adf4bd1d03e9… commit: 66cde7c51a871a86cf8667adf4bd1d03e9b98eb1 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2022-03-29T14:45:47-07:00 summary: bpo-42340: Document issues around KeyboardInterrupt (GH-23255) Update documentation to note that in some circumstances, KeyboardInterrupt may cause code to enter an inconsistent state. Also document sample workaround to avoid KeyboardInterrupt, if needed. (cherry picked from commit d0906c90fcfbc4cfb9bb963eaa6bb152dd543b56) Co-authored-by: benfogle <benfogle(a)gmail.com> files: A Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst M Doc/library/exceptions.rst M Doc/library/signal.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index 2f97bb8131d78..50a3ff3452c99 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -246,6 +246,15 @@ The following exceptions are the exceptions that are usually raised. accidentally caught by code that catches :exc:`Exception` and thus prevent the interpreter from exiting. + .. note:: + + Catching a :exc:`KeyboardInterrupt` requires special consideration. + Because it can be raised at unpredictable points, it may, in some + circumstances, leave the running program in an inconsistent state. It is + generally best to allow :exc:`KeyboardInterrupt` to end the program as + quickly as possible or avoid raising it entirely. (See + :ref:`handlers-and-exceptions`.) + .. exception:: MemoryError diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index 84a569d03eb29..11bdfc8e82831 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -46,6 +46,9 @@ This has consequences: arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes. +* If the handler raises an exception, it will be raised "out of thin air" in + the main thread. See the :ref:`note below <handlers-and-exceptions>` for a + discussion. .. _signals-and-threads: @@ -677,3 +680,70 @@ Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in order to avoid :exc:`BrokenPipeError`. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. + +.. _handlers-and-exceptions: + +Note on Signal Handlers and Exceptions +-------------------------------------- + +If a signal handler raises an exception, the exception will be propagated to +the main thread and may be raised after any :term:`bytecode` instruction. Most +notably, a :exc:`KeyboardInterrupt` may appear at any point during execution. +Most Python code, including the standard library, cannot be made robust against +this, and so a :exc:`KeyboardInterrupt` (or any other exception resulting from +a signal handler) may on rare occasions put the program in an unexpected state. + +To illustrate this issue, consider the following code:: + + class SpamContext: + def __init__(self): + self.lock = threading.Lock() + + def __enter__(self): + # If KeyboardInterrupt occurs here, everything is fine + self.lock.acquire() + # If KeyboardInterrupt occcurs here, __exit__ will not be called + ... + # KeyboardInterrupt could occur just before the function returns + + def __exit__(self, exc_type, exc_val, exc_tb): + ... + self.lock.release() + +For many programs, especially those that merely want to exit on +:exc:`KeyboardInterrupt`, this is not a problem, but applications that are +complex or require high reliability should avoid raising exceptions from signal +handlers. They should also avoid catching :exc:`KeyboardInterrupt` as a means +of gracefully shutting down. Instead, they should install their own +:const:`SIGINT` handler. Below is an example of an HTTP server that avoids +:exc:`KeyboardInterrupt`:: + + import signal + import socket + from selectors import DefaultSelector, EVENT_READ + from http.server import HTTPServer, SimpleHTTPRequestHandler + + interrupt_read, interrupt_write = socket.socketpair() + + def handler(signum, frame): + print('Signal handler called with signal', signum) + interrupt_write.send(b'\0') + signal.signal(signal.SIGINT, handler) + + def serve_forever(httpd): + sel = DefaultSelector() + sel.register(interrupt_read, EVENT_READ) + sel.register(httpd, EVENT_READ) + + while True: + for key, _ in sel.select(): + if key.fileobj == interrupt_read: + interrupt_read.recv(1) + return + if key.fileobj == httpd: + httpd.handle_request() + + print("Serving on port 8000") + httpd = HTTPServer(('', 8000), SimpleHTTPRequestHandler) + serve_forever(httpd) + print("Shutdown...") diff --git a/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst new file mode 100644 index 0000000000000..aa6857497383c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst @@ -0,0 +1,3 @@ +Document that in some circumstances :exc:`KeyboardInterrupt` may cause the +code to enter an inconsistent state. Provided a sample workaround to avoid +it if needed.

1 0

asyncio.Task: rename internal nested variable to don't hide another declaration from outer scope (GH-32181)
by asvetlov 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/a5ba445322f723e04a3e901365512360a9… commit: a5ba445322f723e04a3e901365512360a99c54d4 branch: main author: Andrew Svetlov <andrew.svetlov(a)gmail.com> committer: asvetlov <andrew.svetlov(a)gmail.com> date: 2022-03-30T00:33:51+03:00 summary: asyncio.Task: rename internal nested variable to don't hide another declaration from outer scope (GH-32181) files: M Modules/_asynciomodule.c diff --git a/Modules/_asynciomodule.c b/Modules/_asynciomodule.c index 5b8555e74b375..632a4465c224a 100644 --- a/Modules/_asynciomodule.c +++ b/Modules/_asynciomodule.c @@ -2707,22 +2707,22 @@ task_step_impl(TaskObj *task, PyObject *exc) /* The error is StopIteration and that means that the underlying coroutine has resolved */ - PyObject *res; + PyObject *tmp; if (task->task_must_cancel) { // Task is cancelled right before coro stops. task->task_must_cancel = 0; - res = future_cancel((FutureObj*)task, task->task_cancel_msg); + tmp = future_cancel((FutureObj*)task, task->task_cancel_msg); } else { - res = future_set_result((FutureObj*)task, result); + tmp = future_set_result((FutureObj*)task, result); } Py_DECREF(result); - if (res == NULL) { + if (tmp == NULL) { return NULL; } - Py_DECREF(res); + Py_DECREF(tmp); Py_RETURN_NONE; } @@ -2786,7 +2786,7 @@ task_step_impl(TaskObj *task, PyObject *exc) /* Check if `result` is FutureObj or TaskObj (and not a subclass) */ if (Future_CheckExact(result) || Task_CheckExact(result)) { PyObject *wrapper; - PyObject *res; + PyObject *tmp; FutureObj *fut = (FutureObj*)result; /* Check if `result` future is attached to a different loop */ @@ -2805,13 +2805,13 @@ task_step_impl(TaskObj *task, PyObject *exc) if (wrapper == NULL) { goto fail; } - res = future_add_done_callback( + tmp = future_add_done_callback( (FutureObj*)result, wrapper, task->task_context); Py_DECREF(wrapper); - if (res == NULL) { + if (tmp == NULL) { goto fail; } - Py_DECREF(res); + Py_DECREF(tmp); /* task._fut_waiter = result */ task->task_fut_waiter = result; /* no incref is necessary */ @@ -2853,7 +2853,7 @@ task_step_impl(TaskObj *task, PyObject *exc) if (o != NULL && o != Py_None) { /* `result` is a Future-compatible object */ PyObject *wrapper; - PyObject *res; + PyObject *tmp; int blocking = PyObject_IsTrue(o); Py_DECREF(o); @@ -2897,13 +2897,13 @@ task_step_impl(TaskObj *task, PyObject *exc) PyObject *stack[2]; stack[0] = wrapper; stack[1] = (PyObject *)task->task_context; - res = PyObject_Vectorcall(add_cb, stack, 1, context_kwname); + tmp = PyObject_Vectorcall(add_cb, stack, 1, context_kwname); Py_DECREF(add_cb); Py_DECREF(wrapper); - if (res == NULL) { + if (tmp == NULL) { goto fail; } - Py_DECREF(res); + Py_DECREF(tmp); /* task._fut_waiter = result */ task->task_fut_waiter = result; /* no incref is necessary */

1 0

bpo-33178: Add BigEndianUnion, LittleEndianUnion classes to ctypes (GH-25480)
by gpshead 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/dc2d8404a3ab6288ce112c71da8c65c34c… commit: dc2d8404a3ab6288ce112c71da8c65c34cd3087e branch: main author: Dave Goncalves <davegoncalves(a)gmail.com> committer: gpshead <greg(a)krypto.org> date: 2022-03-29T14:26:27-07:00 summary: bpo-33178: Add BigEndianUnion, LittleEndianUnion classes to ctypes (GH-25480) * bpo-33178: Add BigEndianUnion, LittleEndianUnion classes to ctypes * GH-25480: remove trailing whitespace in ctypes doc * GH-25480: add news entry blurb * GH-25480: corrected formatting error in news blurb * GH-25480: simplified, corrected formatting in news blurb * GH-25480: remove trailing whitespace in news blurb * GH-25480: fixed class markup in news blurb * GH-25480: fixed unsupported type tests and naming per review comments * GH-25480: fixed whitepace errors * condensed base class selection for unsupported byte order tests * added versionadded tags for new EndianUnion classes files: A Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst M Doc/library/ctypes.rst M Lib/ctypes/__init__.py M Lib/ctypes/_endian.py M Lib/ctypes/test/test_byteswap.py diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index dca4c74bab771..6cca569a3f1c8 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -2390,6 +2390,18 @@ Structured data types Abstract base class for unions in native byte order. +.. class:: BigEndianUnion(*args, **kw) + + Abstract base class for unions in *big endian* byte order. + + .. versionadded:: 3.11 + +.. class:: LittleEndianUnion(*args, **kw) + + Abstract base class for unions in *little endian* byte order. + + .. versionadded:: 3.11 + .. class:: BigEndianStructure(*args, **kw) Abstract base class for structures in *big endian* byte order. @@ -2399,8 +2411,8 @@ Structured data types Abstract base class for structures in *little endian* byte order. -Structures with non-native byte order cannot contain pointer type fields, or any -other data types containing pointer type fields. +Structures and unions with non-native byte order cannot contain pointer type +fields, or any other data types containing pointer type fields. .. class:: Structure(*args, **kw) diff --git a/Lib/ctypes/__init__.py b/Lib/ctypes/__init__.py index ab4d31b0acb00..26135ad96296a 100644 --- a/Lib/ctypes/__init__.py +++ b/Lib/ctypes/__init__.py @@ -548,6 +548,7 @@ def DllCanUnloadNow(): return ccom.DllCanUnloadNow() from ctypes._endian import BigEndianStructure, LittleEndianStructure +from ctypes._endian import BigEndianUnion, LittleEndianUnion # Fill in specifically-sized types c_int8 = c_byte diff --git a/Lib/ctypes/_endian.py b/Lib/ctypes/_endian.py index 37444bd6a7dd6..34dee64b1a65a 100644 --- a/Lib/ctypes/_endian.py +++ b/Lib/ctypes/_endian.py @@ -20,7 +20,7 @@ def _other_endian(typ): return typ raise TypeError("This type does not support other endian: %s" % typ) -class _swapped_meta(type(Structure)): +class _swapped_meta: def __setattr__(self, attrname, value): if attrname == "_fields_": fields = [] @@ -31,6 +31,8 @@ def __setattr__(self, attrname, value): fields.append((name, _other_endian(typ)) + rest) value = fields super().__setattr__(attrname, value) +class _swapped_struct_meta(_swapped_meta, type(Structure)): pass +class _swapped_union_meta(_swapped_meta, type(Union)): pass ################################################################ @@ -43,19 +45,34 @@ def __setattr__(self, attrname, value): LittleEndianStructure = Structure - class BigEndianStructure(Structure, metaclass=_swapped_meta): + class BigEndianStructure(Structure, metaclass=_swapped_struct_meta): """Structure with big endian byte order""" __slots__ = () _swappedbytes_ = None + LittleEndianUnion = Union + + class BigEndianUnion(Union, metaclass=_swapped_union_meta): + """Union with big endian byte order""" + __slots__ = () + _swappedbytes_ = None + elif sys.byteorder == "big": _OTHER_ENDIAN = "__ctype_le__" BigEndianStructure = Structure - class LittleEndianStructure(Structure, metaclass=_swapped_meta): + + class LittleEndianStructure(Structure, metaclass=_swapped_struct_meta): """Structure with little endian byte order""" __slots__ = () _swappedbytes_ = None + BigEndianUnion = Union + + class LittleEndianUnion(Union, metaclass=_swapped_union_meta): + """Union with little endian byte order""" + __slots__ = () + _swappedbytes_ = None + else: raise RuntimeError("Invalid byteorder") diff --git a/Lib/ctypes/test/test_byteswap.py b/Lib/ctypes/test/test_byteswap.py index 01c97e83ca7af..7e98559dfbccb 100644 --- a/Lib/ctypes/test/test_byteswap.py +++ b/Lib/ctypes/test/test_byteswap.py @@ -170,40 +170,34 @@ def test_endian_other(self): self.assertIs(c_char.__ctype_le__, c_char) self.assertIs(c_char.__ctype_be__, c_char) - def test_struct_fields_1(self): - if sys.byteorder == "little": - base = BigEndianStructure - else: - base = LittleEndianStructure - - class T(base): - pass - _fields_ = [("a", c_ubyte), - ("b", c_byte), - ("c", c_short), - ("d", c_ushort), - ("e", c_int), - ("f", c_uint), - ("g", c_long), - ("h", c_ulong), - ("i", c_longlong), - ("k", c_ulonglong), - ("l", c_float), - ("m", c_double), - ("n", c_char), - - ("b1", c_byte, 3), - ("b2", c_byte, 3), - ("b3", c_byte, 2), - ("a", c_int * 3 * 3 * 3)] - T._fields_ = _fields_ + def test_struct_fields_unsupported_byte_order(self): + + fields = [ + ("a", c_ubyte), + ("b", c_byte), + ("c", c_short), + ("d", c_ushort), + ("e", c_int), + ("f", c_uint), + ("g", c_long), + ("h", c_ulong), + ("i", c_longlong), + ("k", c_ulonglong), + ("l", c_float), + ("m", c_double), + ("n", c_char), + ("b1", c_byte, 3), + ("b2", c_byte, 3), + ("b3", c_byte, 2), + ("a", c_int * 3 * 3 * 3) + ] # these fields do not support different byte order: for typ in c_wchar, c_void_p, POINTER(c_int): - _fields_.append(("x", typ)) - class T(base): - pass - self.assertRaises(TypeError, setattr, T, "_fields_", [("x", typ)]) + with self.assertRaises(TypeError): + class T(BigEndianStructure if sys.byteorder == "little" else LittleEndianStructure): + _fields_ = fields + [("x", typ)] + def test_struct_struct(self): # nested structures with different byteorders @@ -233,7 +227,7 @@ class TestStructure(parent): self.assertEqual(s.point.x, 1) self.assertEqual(s.point.y, 2) - def test_struct_fields_2(self): + def test_struct_field_alignment(self): # standard packing in struct uses no alignment. # So, we have to align using pad bytes. # @@ -267,7 +261,6 @@ def test_unaligned_nonnative_struct_fields(self): class S(base): _pack_ = 1 _fields_ = [("b", c_byte), - ("h", c_short), ("_1", c_byte), @@ -311,5 +304,61 @@ class S(Structure): s2 = struct.pack(fmt, 0x12, 0x1234, 0x12345678, 3.14) self.assertEqual(bin(s1), bin(s2)) + def test_union_fields_unsupported_byte_order(self): + + fields = [ + ("a", c_ubyte), + ("b", c_byte), + ("c", c_short), + ("d", c_ushort), + ("e", c_int), + ("f", c_uint), + ("g", c_long), + ("h", c_ulong), + ("i", c_longlong), + ("k", c_ulonglong), + ("l", c_float), + ("m", c_double), + ("n", c_char), + ("b1", c_byte, 3), + ("b2", c_byte, 3), + ("b3", c_byte, 2), + ("a", c_int * 3 * 3 * 3) + ] + + # these fields do not support different byte order: + for typ in c_wchar, c_void_p, POINTER(c_int): + with self.assertRaises(TypeError): + class T(BigEndianUnion if sys.byteorder == "little" else LittleEndianUnion): + _fields_ = fields + [("x", typ)] + + def test_union_struct(self): + # nested structures in unions with different byteorders + + # create nested structures in unions with given byteorders and set memory to data + + for nested, data in ( + (BigEndianStructure, b'\0\0\0\1\0\0\0\2'), + (LittleEndianStructure, b'\1\0\0\0\2\0\0\0'), + ): + for parent in ( + BigEndianUnion, + LittleEndianUnion, + Union, + ): + class NestedStructure(nested): + _fields_ = [("x", c_uint32), + ("y", c_uint32)] + + class TestUnion(parent): + _fields_ = [("point", NestedStructure)] + + self.assertEqual(len(data), sizeof(TestUnion)) + ptr = POINTER(TestUnion) + s = cast(data, ptr)[0] + del ctypes._pointer_type_cache[TestUnion] + self.assertEqual(s.point.x, 1) + self.assertEqual(s.point.y, 2) + if __name__ == "__main__": unittest.main() diff --git a/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst b/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst new file mode 100644 index 0000000000000..3646e4a707d6a --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-04-20-16-48-07.bpo-33178.kSnWwb.rst @@ -0,0 +1 @@ +Added :class:`ctypes.BigEndianUnion` and :class:`ctypes.LittleEndianUnion` classes, as originally documented in the library docs but not yet implemented.

1 0

Update glossary.rst (GH-32093)
by asvetlov 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/654bd2152d82b83d7dc037d6d83d60855e… commit: 654bd2152d82b83d7dc037d6d83d60855e3041b7 branch: main author: Géry Ogam <gery.ogam(a)gmail.com> committer: asvetlov <andrew.svetlov(a)gmail.com> date: 2022-03-30T00:21:56+03:00 summary: Update glossary.rst (GH-32093) files: M Doc/glossary.rst diff --git a/Doc/glossary.rst b/Doc/glossary.rst index f0f33d577374b..258a06f721703 100644 --- a/Doc/glossary.rst +++ b/Doc/glossary.rst @@ -628,7 +628,7 @@ Glossary and :class:`tuple`) and some non-sequence types like :class:`dict`, :term:`file objects <file object>`, and objects of any classes you define with an :meth:`__iter__` method or with a :meth:`__getitem__` method - that implements :term:`Sequence <sequence>` semantics. + that implements :term:`sequence` semantics. Iterables can be used in a :keyword:`for` loop and in many other places where a sequence is @@ -680,9 +680,8 @@ Glossary :meth:`str.lower` method can serve as a key function for case insensitive sorts. Alternatively, a key function can be built from a :keyword:`lambda` expression such as ``lambda r: (r[0], r[2])``. Also, - the :mod:`operator` module provides three key function constructors: - :func:`~operator.attrgetter`, :func:`~operator.itemgetter`, and - :func:`~operator.methodcaller`. See the :ref:`Sorting HOW TO + :func:`operator.attrgetter`, :func:`operator.itemgetter`, and + :func:`operator.methodcaller` are three key function constructors. See the :ref:`Sorting HOW TO <sortinghowto>` for examples of how to create and use key functions. keyword argument @@ -743,8 +742,8 @@ Glossary mapping A container object that supports arbitrary key lookups and implements the - methods specified in the :class:`~collections.abc.Mapping` or - :class:`~collections.abc.MutableMapping` + methods specified in the :class:`collections.abc.Mapping` or + :class:`collections.abc.MutableMapping` :ref:`abstract base classes <collections-abstract-base-classes>`. Examples include :class:`dict`, :class:`collections.defaultdict`, :class:`collections.OrderedDict` and :class:`collections.Counter`. @@ -1049,8 +1048,8 @@ Glossary The number of references to an object. When the reference count of an object drops to zero, it is deallocated. Reference counting is generally not visible to Python code, but it is a key element of the - :term:`CPython` implementation. The :mod:`sys` module defines a - :func:`~sys.getrefcount` function that programmers can call to return the + :term:`CPython` implementation. Programmers can call the + :func:`sys.getrefcount` function to return the reference count for a particular object. regular package

1 0

bpo-42340: Document issues around KeyboardInterrupt (GH-23255)
by JelleZijlstra 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/d0906c90fcfbc4cfb9bb963eaa6bb152dd… commit: d0906c90fcfbc4cfb9bb963eaa6bb152dd543b56 branch: main author: benfogle <benfogle(a)gmail.com> committer: JelleZijlstra <jelle.zijlstra(a)gmail.com> date: 2022-03-29T14:21:36-07:00 summary: bpo-42340: Document issues around KeyboardInterrupt (GH-23255) Update documentation to note that in some circumstances, KeyboardInterrupt may cause code to enter an inconsistent state. Also document sample workaround to avoid KeyboardInterrupt, if needed. files: A Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst M Doc/library/exceptions.rst M Doc/library/signal.rst diff --git a/Doc/library/exceptions.rst b/Doc/library/exceptions.rst index d0b2e62c34c96..9e001c8dcd757 100644 --- a/Doc/library/exceptions.rst +++ b/Doc/library/exceptions.rst @@ -254,6 +254,15 @@ The following exceptions are the exceptions that are usually raised. accidentally caught by code that catches :exc:`Exception` and thus prevent the interpreter from exiting. + .. note:: + + Catching a :exc:`KeyboardInterrupt` requires special consideration. + Because it can be raised at unpredictable points, it may, in some + circumstances, leave the running program in an inconsistent state. It is + generally best to allow :exc:`KeyboardInterrupt` to end the program as + quickly as possible or avoid raising it entirely. (See + :ref:`handlers-and-exceptions`.) + .. exception:: MemoryError diff --git a/Doc/library/signal.rst b/Doc/library/signal.rst index abc30362da9e2..c276b52ca4d20 100644 --- a/Doc/library/signal.rst +++ b/Doc/library/signal.rst @@ -46,6 +46,9 @@ This has consequences: arbitrary amount of time, regardless of any signals received. The Python signal handlers will be called when the calculation finishes. +* If the handler raises an exception, it will be raised "out of thin air" in + the main thread. See the :ref:`note below <handlers-and-exceptions>` for a + discussion. .. _signals-and-threads: @@ -712,3 +715,70 @@ Do not set :const:`SIGPIPE`'s disposition to :const:`SIG_DFL` in order to avoid :exc:`BrokenPipeError`. Doing that would cause your program to exit unexpectedly also whenever any socket connection is interrupted while your program is still writing to it. + +.. _handlers-and-exceptions: + +Note on Signal Handlers and Exceptions +-------------------------------------- + +If a signal handler raises an exception, the exception will be propagated to +the main thread and may be raised after any :term:`bytecode` instruction. Most +notably, a :exc:`KeyboardInterrupt` may appear at any point during execution. +Most Python code, including the standard library, cannot be made robust against +this, and so a :exc:`KeyboardInterrupt` (or any other exception resulting from +a signal handler) may on rare occasions put the program in an unexpected state. + +To illustrate this issue, consider the following code:: + + class SpamContext: + def __init__(self): + self.lock = threading.Lock() + + def __enter__(self): + # If KeyboardInterrupt occurs here, everything is fine + self.lock.acquire() + # If KeyboardInterrupt occcurs here, __exit__ will not be called + ... + # KeyboardInterrupt could occur just before the function returns + + def __exit__(self, exc_type, exc_val, exc_tb): + ... + self.lock.release() + +For many programs, especially those that merely want to exit on +:exc:`KeyboardInterrupt`, this is not a problem, but applications that are +complex or require high reliability should avoid raising exceptions from signal +handlers. They should also avoid catching :exc:`KeyboardInterrupt` as a means +of gracefully shutting down. Instead, they should install their own +:const:`SIGINT` handler. Below is an example of an HTTP server that avoids +:exc:`KeyboardInterrupt`:: + + import signal + import socket + from selectors import DefaultSelector, EVENT_READ + from http.server import HTTPServer, SimpleHTTPRequestHandler + + interrupt_read, interrupt_write = socket.socketpair() + + def handler(signum, frame): + print('Signal handler called with signal', signum) + interrupt_write.send(b'\0') + signal.signal(signal.SIGINT, handler) + + def serve_forever(httpd): + sel = DefaultSelector() + sel.register(interrupt_read, EVENT_READ) + sel.register(httpd, EVENT_READ) + + while True: + for key, _ in sel.select(): + if key.fileobj == interrupt_read: + interrupt_read.recv(1) + return + if key.fileobj == httpd: + httpd.handle_request() + + print("Serving on port 8000") + httpd = HTTPServer(('', 8000), SimpleHTTPRequestHandler) + serve_forever(httpd) + print("Shutdown...") diff --git a/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst new file mode 100644 index 0000000000000..aa6857497383c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-11-12-21-26-31.bpo-42340.apumUL.rst @@ -0,0 +1,3 @@ +Document that in some circumstances :exc:`KeyboardInterrupt` may cause the +code to enter an inconsistent state. Provided a sample workaround to avoid +it if needed.

1 0

bpo-47138: Ensure Windows docs build uses the same pinned version as other platforms (GH-32182)
by ned-deily 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/d97497bf1cc6abd4a9a67d8da4f00cf13b… commit: d97497bf1cc6abd4a9a67d8da4f00cf13b3af4b3 branch: 3.7 author: Steve Dower <steve.dower(a)python.org> committer: ned-deily <nad(a)python.org> date: 2022-03-29T15:10:57-04:00 summary: bpo-47138: Ensure Windows docs build uses the same pinned version as other platforms (GH-32182) files: M Doc/make.bat diff --git a/Doc/make.bat b/Doc/make.bat index 63e0bd7236475..f081b4d35a205 100644 --- a/Doc/make.bat +++ b/Doc/make.bat @@ -13,7 +13,7 @@ if not defined SPHINXBUILD ( %PYTHON% -c "import sphinx" > nul 2> nul if errorlevel 1 ( echo Installing sphinx with %PYTHON% - %PYTHON% -m pip install sphinx==2.2.0 + %PYTHON% -m pip install sphinx==2.3.1 blurb docutils==0.17.1 Jinja2==3.0.3 if errorlevel 1 exit /B ) set SPHINXBUILD=%PYTHON% -c "import sphinx.cmd.build, sys; sys.exit(sphinx.cmd.build.main())"

1 0

bpo-14265: Adds fully qualified test name to unittest output (GH-32138)
by asvetlov 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/755be9b1505af591b9f2ee424a6525b6c2… commit: 755be9b1505af591b9f2ee424a6525b6c2b65ce9 branch: main author: Sam Ezeh <sam.z.ezeh(a)gmail.com> committer: asvetlov <andrew.svetlov(a)gmail.com> date: 2022-03-29T18:02:09+03:00 summary: bpo-14265: Adds fully qualified test name to unittest output (GH-32138) Co-authored-by: Andrew Svetlov <andrew.svetlov(a)gmail.com> files: A Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst M Doc/library/unittest.rst M Lib/unittest/case.py M Lib/unittest/test/test_result.py M Misc/ACKS diff --git a/Doc/library/unittest.rst b/Doc/library/unittest.rst index 06df8ce3ad69b..9b8b75acce514 100644 --- a/Doc/library/unittest.rst +++ b/Doc/library/unittest.rst @@ -139,9 +139,9 @@ line, the above script produces an output that looks like this:: Passing the ``-v`` option to your test script will instruct :func:`unittest.main` to enable a higher level of verbosity, and produce the following output:: - test_isupper (__main__.TestStringMethods) ... ok - test_split (__main__.TestStringMethods) ... ok - test_upper (__main__.TestStringMethods) ... ok + test_isupper (__main__.TestStringMethods.test_isupper) ... ok + test_split (__main__.TestStringMethods.test_split) ... ok + test_upper (__main__.TestStringMethods.test_upper) ... ok ---------------------------------------------------------------------- Ran 3 tests in 0.001s @@ -565,10 +565,10 @@ Basic skipping looks like this:: This is the output of running the example above in verbose mode:: - test_format (__main__.MyTestCase) ... skipped 'not supported in this library version' - test_nothing (__main__.MyTestCase) ... skipped 'demonstrating skipping' - test_maybe_skipped (__main__.MyTestCase) ... skipped 'external resource not available' - test_windows_support (__main__.MyTestCase) ... skipped 'requires Windows' + test_format (__main__.MyTestCase.test_format) ... skipped 'not supported in this library version' + test_nothing (__main__.MyTestCase.test_nothing) ... skipped 'demonstrating skipping' + test_maybe_skipped (__main__.MyTestCase.test_maybe_skipped) ... skipped 'external resource not available' + test_windows_support (__main__.MyTestCase.test_windows_support) ... skipped 'requires Windows' ---------------------------------------------------------------------- Ran 4 tests in 0.005s @@ -661,27 +661,33 @@ For example, the following test:: will produce the following output:: ====================================================================== - FAIL: test_even (__main__.NumbersTest) (i=1) + FAIL: test_even (__main__.NumbersTest.test_even) (i=1) + Test that numbers between 0 and 5 are all even. ---------------------------------------------------------------------- Traceback (most recent call last): - File "subtests.py", line 32, in test_even + File "subtests.py", line 11, in test_even self.assertEqual(i % 2, 0) + ^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 0 ====================================================================== - FAIL: test_even (__main__.NumbersTest) (i=3) + FAIL: test_even (__main__.NumbersTest.test_even) (i=3) + Test that numbers between 0 and 5 are all even. ---------------------------------------------------------------------- Traceback (most recent call last): - File "subtests.py", line 32, in test_even + File "subtests.py", line 11, in test_even self.assertEqual(i % 2, 0) + ^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 0 ====================================================================== - FAIL: test_even (__main__.NumbersTest) (i=5) + FAIL: test_even (__main__.NumbersTest.test_even) (i=5) + Test that numbers between 0 and 5 are all even. ---------------------------------------------------------------------- Traceback (most recent call last): - File "subtests.py", line 32, in test_even + File "subtests.py", line 11, in test_even self.assertEqual(i % 2, 0) + ^^^^^^^^^^^^^^^^^^^^^^^^^^ AssertionError: 1 != 0 Without using a subtest, execution would stop after the first failure, @@ -689,7 +695,7 @@ and the error would be less easy to diagnose because the value of ``i`` wouldn't be displayed:: ====================================================================== - FAIL: test_even (__main__.NumbersTest) + FAIL: test_even (__main__.NumbersTest.test_even) ---------------------------------------------------------------------- Traceback (most recent call last): File "subtests.py", line 32, in test_even diff --git a/Lib/unittest/case.py b/Lib/unittest/case.py index 0d550204a7687..55770c06d7c5d 100644 --- a/Lib/unittest/case.py +++ b/Lib/unittest/case.py @@ -478,7 +478,7 @@ def __hash__(self): return hash((type(self), self._testMethodName)) def __str__(self): - return "%s (%s)" % (self._testMethodName, strclass(self.__class__)) + return "%s (%s.%s)" % (self._testMethodName, strclass(self.__class__), self._testMethodName) def __repr__(self): return "<%s testMethod=%s>" % \ diff --git a/Lib/unittest/test/test_result.py b/Lib/unittest/test/test_result.py index c616f28dfee92..b0cc3d867d6e5 100644 --- a/Lib/unittest/test/test_result.py +++ b/Lib/unittest/test/test_result.py @@ -423,7 +423,7 @@ def testGetDescriptionWithoutDocstring(self): self.assertEqual( result.getDescription(self), 'testGetDescriptionWithoutDocstring (' + __name__ + - '.Test_TextTestResult)') + '.Test_TextTestResult.testGetDescriptionWithoutDocstring)') def testGetSubTestDescriptionWithoutDocstring(self): with self.subTest(foo=1, bar=2): @@ -431,13 +431,14 @@ def testGetSubTestDescriptionWithoutDocstring(self): self.assertEqual( result.getDescription(self._subtest), 'testGetSubTestDescriptionWithoutDocstring (' + __name__ + - '.Test_TextTestResult) (foo=1, bar=2)') + '.Test_TextTestResult.testGetSubTestDescriptionWithoutDocstring) (foo=1, bar=2)') + with self.subTest('some message'): result = unittest.TextTestResult(None, True, 1) self.assertEqual( result.getDescription(self._subtest), 'testGetSubTestDescriptionWithoutDocstring (' + __name__ + - '.Test_TextTestResult) [some message]') + '.Test_TextTestResult.testGetSubTestDescriptionWithoutDocstring) [some message]') def testGetSubTestDescriptionWithoutDocstringAndParams(self): with self.subTest(): @@ -445,10 +446,11 @@ def testGetSubTestDescriptionWithoutDocstringAndParams(self): self.assertEqual( result.getDescription(self._subtest), 'testGetSubTestDescriptionWithoutDocstringAndParams ' - '(' + __name__ + '.Test_TextTestResult) (<subtest>)') + '(' + __name__ + '.Test_TextTestResult.testGetSubTestDescriptionWithoutDocstringAndParams) ' + '(<subtest>)') def testGetSubTestDescriptionForFalsyValues(self): - expected = 'testGetSubTestDescriptionForFalsyValues (%s.Test_TextTestResult) [%s]' + expected = 'testGetSubTestDescriptionForFalsyValues (%s.Test_TextTestResult.testGetSubTestDescriptionForFalsyValues) [%s]' result = unittest.TextTestResult(None, True, 1) for arg in [0, None, []]: with self.subTest(arg): @@ -464,7 +466,8 @@ def testGetNestedSubTestDescriptionWithoutDocstring(self): self.assertEqual( result.getDescription(self._subtest), 'testGetNestedSubTestDescriptionWithoutDocstring ' - '(' + __name__ + '.Test_TextTestResult) (baz=2, bar=3, foo=1)') + '(' + __name__ + '.Test_TextTestResult.testGetNestedSubTestDescriptionWithoutDocstring) ' + '(baz=2, bar=3, foo=1)') def testGetDuplicatedNestedSubTestDescriptionWithoutDocstring(self): with self.subTest(foo=1, bar=2): @@ -473,7 +476,7 @@ def testGetDuplicatedNestedSubTestDescriptionWithoutDocstring(self): self.assertEqual( result.getDescription(self._subtest), 'testGetDuplicatedNestedSubTestDescriptionWithoutDocstring ' - '(' + __name__ + '.Test_TextTestResult) (baz=3, bar=4, foo=1)') + '(' + __name__ + '.Test_TextTestResult.testGetDuplicatedNestedSubTestDescriptionWithoutDocstring) (baz=3, bar=4, foo=1)') @unittest.skipIf(sys.flags.optimize >= 2, "Docstrings are omitted with -O2 and above") @@ -483,7 +486,7 @@ def testGetDescriptionWithOneLineDocstring(self): self.assertEqual( result.getDescription(self), ('testGetDescriptionWithOneLineDocstring ' - '(' + __name__ + '.Test_TextTestResult)\n' + '(' + __name__ + '.Test_TextTestResult.testGetDescriptionWithOneLineDocstring)\n' 'Tests getDescription() for a method with a docstring.')) @unittest.skipIf(sys.flags.optimize >= 2, @@ -495,7 +498,9 @@ def testGetSubTestDescriptionWithOneLineDocstring(self): self.assertEqual( result.getDescription(self._subtest), ('testGetSubTestDescriptionWithOneLineDocstring ' - '(' + __name__ + '.Test_TextTestResult) (foo=1, bar=2)\n' + '(' + __name__ + '.Test_TextTestResult.testGetSubTestDescriptionWithOneLineDocstring) ' + '(foo=1, bar=2)\n' + 'Tests getDescription() for a method with a docstring.')) @unittest.skipIf(sys.flags.optimize >= 2, @@ -508,7 +513,7 @@ def testGetDescriptionWithMultiLineDocstring(self): self.assertEqual( result.getDescription(self), ('testGetDescriptionWithMultiLineDocstring ' - '(' + __name__ + '.Test_TextTestResult)\n' + '(' + __name__ + '.Test_TextTestResult.testGetDescriptionWithMultiLineDocstring)\n' 'Tests getDescription() for a method with a longer ' 'docstring.')) @@ -523,7 +528,8 @@ def testGetSubTestDescriptionWithMultiLineDocstring(self): self.assertEqual( result.getDescription(self._subtest), ('testGetSubTestDescriptionWithMultiLineDocstring ' - '(' + __name__ + '.Test_TextTestResult) (foo=1, bar=2)\n' + '(' + __name__ + '.Test_TextTestResult.testGetSubTestDescriptionWithMultiLineDocstring) ' + '(foo=1, bar=2)\n' 'Tests getDescription() for a method with a longer ' 'docstring.')) @@ -582,17 +588,17 @@ def testDotsOutput(self): def testLongOutput(self): classname = f'{__name__}.{self.Test.__qualname__}' self.assertEqual(self._run_test('testSuccess', 2), - f'testSuccess ({classname}) ... ok\n') + f'testSuccess ({classname}.testSuccess) ... ok\n') self.assertEqual(self._run_test('testSkip', 2), - f"testSkip ({classname}) ... skipped 'skip'\n") + f"testSkip ({classname}.testSkip) ... skipped 'skip'\n") self.assertEqual(self._run_test('testFail', 2), - f'testFail ({classname}) ... FAIL\n') + f'testFail ({classname}.testFail) ... FAIL\n') self.assertEqual(self._run_test('testError', 2), - f'testError ({classname}) ... ERROR\n') + f'testError ({classname}.testError) ... ERROR\n') self.assertEqual(self._run_test('testExpectedFailure', 2), - f'testExpectedFailure ({classname}) ... expected failure\n') + f'testExpectedFailure ({classname}.testExpectedFailure) ... expected failure\n') self.assertEqual(self._run_test('testUnexpectedSuccess', 2), - f'testUnexpectedSuccess ({classname}) ... unexpected success\n') + f'testUnexpectedSuccess ({classname}.testUnexpectedSuccess) ... unexpected success\n') def testDotsOutputSubTestSuccess(self): self.assertEqual(self._run_test('testSubTestSuccess', 1), '.') @@ -600,7 +606,7 @@ def testDotsOutputSubTestSuccess(self): def testLongOutputSubTestSuccess(self): classname = f'{__name__}.{self.Test.__qualname__}' self.assertEqual(self._run_test('testSubTestSuccess', 2), - f'testSubTestSuccess ({classname}) ... ok\n') + f'testSubTestSuccess ({classname}.testSubTestSuccess) ... ok\n') def testDotsOutputSubTestMixed(self): self.assertEqual(self._run_test('testSubTestMixed', 1), 'sFE') @@ -608,10 +614,10 @@ def testDotsOutputSubTestMixed(self): def testLongOutputSubTestMixed(self): classname = f'{__name__}.{self.Test.__qualname__}' self.assertEqual(self._run_test('testSubTestMixed', 2), - f'testSubTestMixed ({classname}) ... \n' - f" testSubTestMixed ({classname}) [skip] (b=2) ... skipped 'skip'\n" - f' testSubTestMixed ({classname}) [fail] (c=3) ... FAIL\n' - f' testSubTestMixed ({classname}) [error] (d=4) ... ERROR\n') + f'testSubTestMixed ({classname}.testSubTestMixed) ... \n' + f" testSubTestMixed ({classname}.testSubTestMixed) [skip] (b=2) ... skipped 'skip'\n" + f' testSubTestMixed ({classname}.testSubTestMixed) [fail] (c=3) ... FAIL\n' + f' testSubTestMixed ({classname}.testSubTestMixed) [error] (d=4) ... ERROR\n') def testDotsOutputTearDownFail(self): out = self._run_test('testSuccess', 1, AssertionError('fail')) @@ -627,19 +633,19 @@ def testLongOutputTearDownFail(self): classname = f'{__name__}.{self.Test.__qualname__}' out = self._run_test('testSuccess', 2, AssertionError('fail')) self.assertEqual(out, - f'testSuccess ({classname}) ... FAIL\n') + f'testSuccess ({classname}.testSuccess) ... FAIL\n') out = self._run_test('testError', 2, AssertionError('fail')) self.assertEqual(out, - f'testError ({classname}) ... ERROR\n' - f'testError ({classname}) ... FAIL\n') + f'testError ({classname}.testError) ... ERROR\n' + f'testError ({classname}.testError) ... FAIL\n') out = self._run_test('testFail', 2, Exception('error')) self.assertEqual(out, - f'testFail ({classname}) ... FAIL\n' - f'testFail ({classname}) ... ERROR\n') + f'testFail ({classname}.testFail) ... FAIL\n' + f'testFail ({classname}.testFail) ... ERROR\n') out = self._run_test('testSkip', 2, AssertionError('fail')) self.assertEqual(out, - f"testSkip ({classname}) ... skipped 'skip'\n" - f'testSkip ({classname}) ... FAIL\n') + f"testSkip ({classname}.testSkip) ... skipped 'skip'\n" + f'testSkip ({classname}.testSkip) ... FAIL\n') classDict = dict(unittest.TestResult.__dict__) @@ -852,7 +858,7 @@ def test_foo(self): expected_out = '\nStdout:\nset up\n' self.assertEqual(stdout.getvalue(), expected_out) self.assertEqual(len(result.errors), 1) - description = f'test_foo ({strclass(Foo)})' + description = f'test_foo ({strclass(Foo)}.test_foo)' test_case, formatted_exc = result.errors[0] self.assertEqual(str(test_case), description) self.assertIn('ZeroDivisionError: division by zero', formatted_exc) @@ -874,7 +880,7 @@ def test_foo(self): expected_out = '\nStdout:\ntear down\n' self.assertEqual(stdout.getvalue(), expected_out) self.assertEqual(len(result.errors), 1) - description = f'test_foo ({strclass(Foo)})' + description = f'test_foo ({strclass(Foo)}.test_foo)' test_case, formatted_exc = result.errors[0] self.assertEqual(str(test_case), description) self.assertIn('ZeroDivisionError: division by zero', formatted_exc) @@ -897,7 +903,7 @@ def test_foo(self): expected_out = '\nStdout:\nset up\ndo cleanup2\ndo cleanup1\n' self.assertEqual(stdout.getvalue(), expected_out) self.assertEqual(len(result.errors), 2) - description = f'test_foo ({strclass(Foo)})' + description = f'test_foo ({strclass(Foo)}.test_foo)' test_case, formatted_exc = result.errors[0] self.assertEqual(str(test_case), description) self.assertIn('ValueError: bad cleanup2', formatted_exc) @@ -928,7 +934,7 @@ def test_foo(self): expected_out = '\nStdout:\nset up\ndo cleanup2\ndo cleanup1\n' self.assertEqual(stdout.getvalue(), expected_out) self.assertEqual(len(result.errors), 3) - description = f'test_foo ({strclass(Foo)})' + description = f'test_foo ({strclass(Foo)}.test_foo)' test_case, formatted_exc = result.errors[0] self.assertEqual(str(test_case), description) self.assertIn('ZeroDivisionError: division by zero', formatted_exc) @@ -971,7 +977,7 @@ def test_foo(self): expected_out = '\nStdout:\nset up\ntear down\ndo cleanup2\ndo cleanup1\n' self.assertEqual(stdout.getvalue(), expected_out) self.assertEqual(len(result.errors), 3) - description = f'test_foo ({strclass(Foo)})' + description = f'test_foo ({strclass(Foo)}.test_foo)' test_case, formatted_exc = result.errors[0] self.assertEqual(str(test_case), description) self.assertIn('ZeroDivisionError: division by zero', formatted_exc) diff --git a/Misc/ACKS b/Misc/ACKS index efa2474c3cfdb..72e2cfe13775a 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -516,6 +516,7 @@ Daniel Evers evilzero Winston Ewert Greg Ewing +Sam Ezeh Martijn Faassen Clovis Fabricio Andreas Faerber diff --git a/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst b/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst new file mode 100644 index 0000000000000..308ce36c6ba50 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2022-03-27-10-41-24.bpo-14265.OBMlAi.rst @@ -0,0 +1 @@ +Adds the fully qualified test name to unittest output

1 0

bpo-35859: Fix a few long-standing bugs in re engine (GH-12427)
by serhiy-storchaka 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/356997cccc21a3391175d20e9ef03d4346… commit: 356997cccc21a3391175d20e9ef03d434675b496 branch: main author: Ma Lin <animalize(a)users.noreply.github.com> committer: serhiy-storchaka <storchaka(a)gmail.com> date: 2022-03-29T17:31:01+03:00 summary: bpo-35859: Fix a few long-standing bugs in re engine (GH-12427) In rare cases, capturing group could get wrong result. Regular expression engines in Perl and Java have similar bugs. The new behavior now matches the behavior of more modern RE engines: in the regex module and in PHP, Ruby and Node.js. files: A Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst M Doc/whatsnew/3.11.rst M Lib/test/test_re.py M Modules/_sre.c M Modules/sre_lib.h diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 41e4659b0f779..837d8c8cbd016 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -718,6 +718,11 @@ Changes in the Python API deprecated since Python 3.6. (Contributed by Serhiy Storchaka in :issue:`47066`.) +* :mod:`re` module: Fix a few long-standing bugs where, in rare cases, + capturing group could get wrong result. So the result may be different than + before. + (Contributed by Ma Lin in :issue:`35859`.) + * The *population* parameter of :func:`random.sample` must be a sequence. Automatic conversion of sets to lists is no longer supported. If the sample size is larger than the population size, a :exc:`ValueError` is raised. diff --git a/Lib/test/test_re.py b/Lib/test/test_re.py index fd6db6a300d03..85716fbe2a8e8 100644 --- a/Lib/test/test_re.py +++ b/Lib/test/test_re.py @@ -2033,6 +2033,75 @@ def test_bug_34294(self): {'tag': 'foo', 'text': None}, {'tag': 'foo', 'text': None}]) + def test_MARK_PUSH_macro_bug(self): + # issue35859, MARK_PUSH() macro didn't protect MARK-0 if it + # was the only available mark. + self.assertEqual(re.match(r'(ab|a)*?b', 'ab').groups(), ('a',)) + self.assertEqual(re.match(r'(ab|a)+?b', 'ab').groups(), ('a',)) + self.assertEqual(re.match(r'(ab|a){0,2}?b', 'ab').groups(), ('a',)) + self.assertEqual(re.match(r'(.b|a)*?b', 'ab').groups(), ('a',)) + + def test_MIN_UNTIL_mark_bug(self): + # Fixed in issue35859, reported in issue9134. + # JUMP_MIN_UNTIL_2 should MARK_PUSH() if in a repeat + s = 'axxzbcz' + p = r'(?:(?:a|bc)*?(xx)??z)*' + self.assertEqual(re.match(p, s).groups(), ('xx',)) + + # test-case provided by issue9134 + s = 'xtcxyzxc' + p = r'((x|yz)+?(t)??c)*' + m = re.match(p, s) + self.assertEqual(m.span(), (0, 8)) + self.assertEqual(m.span(2), (6, 7)) + self.assertEqual(m.groups(), ('xyzxc', 'x', 't')) + + def test_REPEAT_ONE_mark_bug(self): + # issue35859 + # JUMP_REPEAT_ONE_1 should MARK_PUSH() if in a repeat + s = 'aabaab' + p = r'(?:[^b]*a(?=(b)|(a))ab)*' + m = re.match(p, s) + self.assertEqual(m.span(), (0, 6)) + self.assertEqual(m.span(2), (4, 5)) + self.assertEqual(m.groups(), (None, 'a')) + + # JUMP_REPEAT_ONE_2 should MARK_PUSH() if in a repeat + s = 'abab' + p = r'(?:[^b]*(?=(b)|(a))ab)*' + m = re.match(p, s) + self.assertEqual(m.span(), (0, 4)) + self.assertEqual(m.span(2), (2, 3)) + self.assertEqual(m.groups(), (None, 'a')) + + self.assertEqual(re.match(r'(ab?)*?b', 'ab').groups(), ('a',)) + + def test_MIN_REPEAT_ONE_mark_bug(self): + # issue35859 + # JUMP_MIN_REPEAT_ONE should MARK_PUSH() if in a repeat + s = 'abab' + p = r'(?:.*?(?=(a)|(b))b)*' + m = re.match(p, s) + self.assertEqual(m.span(), (0, 4)) + self.assertEqual(m.span(2), (3, 4)) + self.assertEqual(m.groups(), (None, 'b')) + + s = 'axxzaz' + p = r'(?:a*?(xx)??z)*' + self.assertEqual(re.match(p, s).groups(), ('xx',)) + + def test_ASSERT_NOT_mark_bug(self): + # Fixed in issue35859, reported in issue725149. + # JUMP_ASSERT_NOT should LASTMARK_SAVE() + self.assertEqual(re.match(r'(?!(..)c)', 'ab').groups(), (None,)) + + # JUMP_ASSERT_NOT should MARK_PUSH() if in a repeat + m = re.match(r'((?!(ab)c)(.))*', 'abab') + self.assertEqual(m.span(), (0, 4)) + self.assertEqual(m.span(1), (3, 4)) + self.assertEqual(m.span(3), (3, 4)) + self.assertEqual(m.groups(), ('b', None, 'b')) + def test_bug_40736(self): with self.assertRaisesRegex(TypeError, "got 'int'"): re.search("x*", 5) diff --git a/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst b/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst new file mode 100644 index 0000000000000..8c88ef01164e4 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-03-14-09-08-25.bpo-35859.8lFdLe.rst @@ -0,0 +1,2 @@ +:mod:`re` module, fix a few bugs about capturing group. In rare cases, +capturing group gets an incorrect string. Patch by Ma Lin. diff --git a/Modules/_sre.c b/Modules/_sre.c index 35bdb4f3d2202..48193f82475a4 100644 --- a/Modules/_sre.c +++ b/Modules/_sre.c @@ -532,6 +532,14 @@ state_getslice(SRE_STATE* state, Py_ssize_t index, PyObject* string, int empty) } else { i = STATE_OFFSET(state, state->mark[index]); j = STATE_OFFSET(state, state->mark[index+1]); + + /* check wrong span */ + if (i > j) { + PyErr_SetString(PyExc_SystemError, + "The span of capturing group is wrong," + " please report a bug for the re module."); + return NULL; + } } return getslice(state->isbytes, state->beginning, string, i, j); @@ -2477,6 +2485,15 @@ pattern_new_match(_sremodulestate* module_state, if (j+1 <= state->lastmark && state->mark[j] && state->mark[j+1]) { match->mark[j+2] = ((char*) state->mark[j] - base) / n; match->mark[j+3] = ((char*) state->mark[j+1] - base) / n; + + /* check wrong span */ + if (match->mark[j+2] > match->mark[j+3]) { + PyErr_SetString(PyExc_SystemError, + "The span of capturing group is wrong," + " please report a bug for the re module."); + Py_DECREF(match); + return NULL; + } } else match->mark[j+2] = match->mark[j+3] = -1; /* undefined */ diff --git a/Modules/sre_lib.h b/Modules/sre_lib.h index a82210ff94a0f..8e4e714eada38 100644 --- a/Modules/sre_lib.h +++ b/Modules/sre_lib.h @@ -449,20 +449,20 @@ do { \ DATA_STACK_LOOKUP_AT(state,t,p,pos) #define MARK_PUSH(lastmark) \ - do if (lastmark > 0) { \ + do if (lastmark >= 0) { \ i = lastmark; /* ctx->lastmark may change if reallocated */ \ DATA_STACK_PUSH(state, state->mark, (i+1)*sizeof(void*)); \ } while (0) #define MARK_POP(lastmark) \ - do if (lastmark > 0) { \ + do if (lastmark >= 0) { \ DATA_STACK_POP(state, state->mark, (lastmark+1)*sizeof(void*), 1); \ } while (0) #define MARK_POP_KEEP(lastmark) \ - do if (lastmark > 0) { \ + do if (lastmark >= 0) { \ DATA_STACK_POP(state, state->mark, (lastmark+1)*sizeof(void*), 0); \ } while (0) #define MARK_POP_DISCARD(lastmark) \ - do if (lastmark > 0) { \ + do if (lastmark >= 0) { \ DATA_STACK_POP_DISCARD(state, (lastmark+1)*sizeof(void*)); \ } while (0) @@ -770,8 +770,7 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) /* <BRANCH> <0=skip> code <JUMP> ... <NULL> */ TRACE(("|%p|%p|BRANCH\n", ctx->pattern, ctx->ptr)); LASTMARK_SAVE(); - ctx->u.rep = state->repeat; - if (ctx->u.rep) + if (state->repeat) MARK_PUSH(ctx->lastmark); for (; ctx->pattern[0]; ctx->pattern += ctx->pattern[0]) { if (ctx->pattern[1] == SRE_OP_LITERAL && @@ -786,16 +785,16 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) state->ptr = ctx->ptr; DO_JUMP(JUMP_BRANCH, jump_branch, ctx->pattern+1); if (ret) { - if (ctx->u.rep) + if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_SUCCESS; } - if (ctx->u.rep) + if (state->repeat) MARK_POP_KEEP(ctx->lastmark); LASTMARK_RESTORE(); } - if (ctx->u.rep) + if (state->repeat) MARK_POP_DISCARD(ctx->lastmark); RETURN_FAILURE; @@ -841,6 +840,8 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) } LASTMARK_SAVE(); + if (state->repeat) + MARK_PUSH(ctx->lastmark); if (ctx->pattern[ctx->pattern[0]] == SRE_OP_LITERAL) { /* tail starts with a literal. skip positions where @@ -858,16 +859,20 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) DO_JUMP(JUMP_REPEAT_ONE_1, jump_repeat_one_1, ctx->pattern+ctx->pattern[0]); if (ret) { + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_SUCCESS; } - + if (state->repeat) + MARK_POP_KEEP(ctx->lastmark); LASTMARK_RESTORE(); ctx->ptr--; ctx->count--; } - + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); } else { /* general case */ while (ctx->count >= (Py_ssize_t) ctx->pattern[1]) { @@ -875,13 +880,20 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) DO_JUMP(JUMP_REPEAT_ONE_2, jump_repeat_one_2, ctx->pattern+ctx->pattern[0]); if (ret) { + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_SUCCESS; } + if (state->repeat) + MARK_POP_KEEP(ctx->lastmark); + LASTMARK_RESTORE(); + ctx->ptr--; ctx->count--; - LASTMARK_RESTORE(); } + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); } RETURN_FAILURE; @@ -930,15 +942,24 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) } else { /* general case */ LASTMARK_SAVE(); + if (state->repeat) + MARK_PUSH(ctx->lastmark); + while ((Py_ssize_t)ctx->pattern[2] == SRE_MAXREPEAT || ctx->count <= (Py_ssize_t)ctx->pattern[2]) { state->ptr = ctx->ptr; DO_JUMP(JUMP_MIN_REPEAT_ONE,jump_min_repeat_one, ctx->pattern+ctx->pattern[0]); if (ret) { + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_SUCCESS; } + if (state->repeat) + MARK_POP_KEEP(ctx->lastmark); + LASTMARK_RESTORE(); + state->ptr = ctx->ptr; ret = SRE(count)(state, ctx->pattern+3, 1); RETURN_ON_ERROR(ret); @@ -948,8 +969,9 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) assert(ret == 1); ctx->ptr++; ctx->count++; - LASTMARK_RESTORE(); } + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); } RETURN_FAILURE; @@ -1098,8 +1120,9 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) tail matches */ state->repeat = ctx->u.rep->prev; DO_JUMP(JUMP_MAX_UNTIL_3, jump_max_until_3, ctx->pattern); + state->repeat = ctx->u.rep; // restore repeat before return + RETURN_ON_SUCCESS(ret); - state->repeat = ctx->u.rep; state->ptr = ctx->ptr; RETURN_FAILURE; @@ -1132,21 +1155,29 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) RETURN_FAILURE; } - LASTMARK_SAVE(); - /* see if the tail matches */ state->repeat = ctx->u.rep->prev; + + LASTMARK_SAVE(); + if (state->repeat) + MARK_PUSH(ctx->lastmark); + DO_JUMP(JUMP_MIN_UNTIL_2, jump_min_until_2, ctx->pattern); + SRE_REPEAT *repeat_of_tail = state->repeat; + state->repeat = ctx->u.rep; // restore repeat before return + if (ret) { + if (repeat_of_tail) + MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_SUCCESS; } + if (repeat_of_tail) + MARK_POP(ctx->lastmark); + LASTMARK_RESTORE(); - state->repeat = ctx->u.rep; state->ptr = ctx->ptr; - LASTMARK_RESTORE(); - if ((ctx->count >= (Py_ssize_t) ctx->u.rep->pattern[2] && ctx->u.rep->pattern[2] != SRE_MAXREPEAT) || state->ptr == ctx->u.rep->last_ptr) @@ -1444,11 +1475,20 @@ SRE(match)(SRE_STATE* state, const SRE_CODE* pattern, int toplevel) ctx->ptr, ctx->pattern[1])); if (ctx->ptr - (SRE_CHAR *)state->beginning >= (Py_ssize_t)ctx->pattern[1]) { state->ptr = ctx->ptr - ctx->pattern[1]; + LASTMARK_SAVE(); + if (state->repeat) + MARK_PUSH(ctx->lastmark); + DO_JUMP0(JUMP_ASSERT_NOT, jump_assert_not, ctx->pattern+2); if (ret) { + if (state->repeat) + MARK_POP_DISCARD(ctx->lastmark); RETURN_ON_ERROR(ret); RETURN_FAILURE; } + if (state->repeat) + MARK_POP(ctx->lastmark); + LASTMARK_RESTORE(); } ctx->pattern += ctx->pattern[0]; break;

1 0

Fix typo in the sqlite3 docs (GH-31915) (GH-32157)
by JelleZijlstra 29 Mar '22

29 Mar '22

https://github.com/python/cpython/commit/370900d7d8ba29eb52e419c3163599b4ac… commit: 370900d7d8ba29eb52e419c3163599b4ac55f121 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: JelleZijlstra <jelle.zijlstra(a)gmail.com> date: 2022-03-28T21:21:27-07:00 summary: Fix typo in the sqlite3 docs (GH-31915) (GH-32157) Co-authored-by: Jonathan <89750679+AHypnotoad(a)users.noreply.github.com> (cherry picked from commit 66584c890d016e40d707400130d1cd98f2aedde9) Co-authored-by: Jonathan <jonathan.joyner94(a)gmail.com> Co-authored-by: Jelle Zijlstra <jelle.zijlstra(a)gmail.com> files: M Lib/sqlite3/__init__.py diff --git a/Lib/sqlite3/__init__.py b/Lib/sqlite3/__init__.py index 0dedf186b1a1e..5a2dbd360fb49 100644 --- a/Lib/sqlite3/__init__.py +++ b/Lib/sqlite3/__init__.py @@ -21,7 +21,7 @@ # 3. This notice may not be removed or altered from any source distribution. """ -The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compilant +The sqlite3 extension module provides a DB-API 2.0 (PEP 249) compliant interface to the SQLite library, and requires SQLite 3.7.15 or newer. To use the module, start by creating a database Connection object:

1 0