Mailman 3 March 2022 - Python-checkins

bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32154)
by ned-deily March 28, 2022

March 28, 2022

https://github.com/python/cpython/commit/9194a7b8990a0feec1209b1e5694df3bf4… commit: 9194a7b8990a0feec1209b1e5694df3bf42906d8 branch: 3.10 author: Hugo van Kemenade <hugovk(a)users.noreply.github.com> committer: ned-deily <nad(a)python.org> date: 2022-03-28T13:06:16-04:00 summary: bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32154) Co-authored-by: Maciej Olko <maciej.olko(a)gmail.com> files: A Misc/NEWS.d/next/Documentation/2022-03-28-12-29-42.bpo-47138.2B4N-k.rst M Doc/requirements.txt diff --git a/Doc/requirements.txt b/Doc/requirements.txt index 95d320f4cb1f3..793df32ecf65f 100644 --- a/Doc/requirements.txt +++ b/Doc/requirements.txt @@ -8,6 +8,8 @@ sphinx==3.2.1 # version 3.2.1. It can be removed after bumping Sphinx version to at # least 3.5.4. docutils==0.17.1 +# Jinja version is pinned to a version compatible with Sphinx version 3.2.1. +jinja2==3.0.3 blurb diff --git a/Misc/NEWS.d/next/Documentation/2022-03-28-12-29-42.bpo-47138.2B4N-k.rst b/Misc/NEWS.d/next/Documentation/2022-03-28-12-29-42.bpo-47138.2B4N-k.rst new file mode 100644 index 0000000000000..e15148b73b23b --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-28-12-29-42.bpo-47138.2B4N-k.rst @@ -0,0 +1 @@ +Pin Jinja to a version compatible with Sphinx version 3.2.1.

1 0

bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32111)
by ned-deily March 28, 2022

March 28, 2022

https://github.com/python/cpython/commit/25f00bfb264a3197ac91c41cdec15036fd… commit: 25f00bfb264a3197ac91c41cdec15036fd8401f1 branch: 3.7 author: m-aciek <maciej.olko(a)gmail.com> committer: ned-deily <nad(a)python.org> date: 2022-03-28T13:05:01-04:00 summary: bpo-47138: Fix documentation build by pinning Jinja version to 3.0.3 (GH-32111) files: A Misc/NEWS.d/next/Documentation/2022-03-28-12-39-20.bpo-47138.TEZRwC.rst M Doc/Makefile diff --git a/Doc/Makefile b/Doc/Makefile index 1393bcb52500e..1b8cbfa479e50 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -127,7 +127,7 @@ clean: venv: $(PYTHON) -m venv $(VENVDIR) - $(VENVDIR)/bin/python3 -m pip install -U Sphinx==2.3.1 blurb docutils==0.17.1 + $(VENVDIR)/bin/python3 -m pip install -U Sphinx==2.3.1 blurb docutils==0.17.1 Jinja2==3.0.3 @echo "The venv has been created in the $(VENVDIR) directory" dist: diff --git a/Misc/NEWS.d/next/Documentation/2022-03-28-12-39-20.bpo-47138.TEZRwC.rst b/Misc/NEWS.d/next/Documentation/2022-03-28-12-39-20.bpo-47138.TEZRwC.rst new file mode 100644 index 0000000000000..aa3ed79d5e29c --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2022-03-28-12-39-20.bpo-47138.TEZRwC.rst @@ -0,0 +1 @@ +Pin Jinja to a version compatible with Sphinx version 2.3.1.

1 0

bpo-47070: Add _PyBytes_Repeat() (GH-31999)
by sweeneyde March 28, 2022

March 28, 2022

https://github.com/python/cpython/commit/850687df47b03e98c1433e6e70e71a8921… commit: 850687df47b03e98c1433e6e70e71a8921eb4454 branch: main author: Pieter Eendebak <pieter.eendebak(a)gmail.com> committer: sweeneyde <36520290+sweeneyde(a)users.noreply.github.com> date: 2022-03-28T04:43:45-04:00 summary: bpo-47070: Add _PyBytes_Repeat() (GH-31999) Use it where appropriate: the repeat functions of `array.array`, `bytes`, `bytearray`, and `str`. files: A Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst M Include/internal/pycore_bytesobject.h M Modules/arraymodule.c M Objects/bytearrayobject.c M Objects/bytesobject.c M Objects/unicodeobject.c diff --git a/Include/internal/pycore_bytesobject.h b/Include/internal/pycore_bytesobject.h index b8061607a518d..9173a4f105f80 100644 --- a/Include/internal/pycore_bytesobject.h +++ b/Include/internal/pycore_bytesobject.h @@ -33,6 +33,19 @@ _PyBytes_ReverseFind(const char *haystack, Py_ssize_t len_haystack, const char *needle, Py_ssize_t len_needle, Py_ssize_t offset); + +/** Helper function to implement the repeat and inplace repeat methods on a buffer + * + * len_dest is assumed to be an integer multiple of len_src. + * If src equals dest, then assume the operation is inplace. + * + * This method repeately doubles the number of bytes copied to reduce + * the number of invocations of memcpy. + */ +PyAPI_FUNC(void) +_PyBytes_Repeat(char* dest, Py_ssize_t len_dest, + const char* src, Py_ssize_t len_src); + #ifdef __cplusplus } #endif diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst new file mode 100644 index 0000000000000..568974f251f32 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-19-21-50-59.bpo-47070.wPcsQh.rst @@ -0,0 +1,3 @@ +Improve performance of ``array_inplace_repeat`` by reducing the number of invocations of ``memcpy``. +Refactor the ``repeat`` and inplace ``repeat`` methods of ``array``, ``bytes``, ``bytearray`` +and ``unicodeobject`` to use the common ``_PyBytes_Repeat``. diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c index 18991f81480d0..a04e6a4e070d9 100644 --- a/Modules/arraymodule.c +++ b/Modules/arraymodule.c @@ -10,6 +10,7 @@ #define PY_SSIZE_T_CLEAN #include "Python.h" #include "pycore_moduleobject.h" // _PyModule_GetState() +#include "pycore_bytesobject.h" // _PyBytes_Repeat #include "structmember.h" // PyMemberDef #include <stddef.h> // offsetof() #include <stddef.h> @@ -910,34 +911,24 @@ static PyObject * array_repeat(arrayobject *a, Py_ssize_t n) { array_state *state = find_array_state_by_type(Py_TYPE(a)); - Py_ssize_t size; - arrayobject *np; - Py_ssize_t oldbytes, newbytes; + if (n < 0) n = 0; - if ((Py_SIZE(a) != 0) && (n > PY_SSIZE_T_MAX / Py_SIZE(a))) { + const Py_ssize_t array_length = Py_SIZE(a); + if ((array_length != 0) && (n > PY_SSIZE_T_MAX / array_length)) { return PyErr_NoMemory(); } - size = Py_SIZE(a) * n; - np = (arrayobject *) newarrayobject(state->ArrayType, size, a->ob_descr); + Py_ssize_t size = array_length * n; + arrayobject* np = (arrayobject *) newarrayobject(state->ArrayType, size, a->ob_descr); if (np == NULL) return NULL; if (size == 0) return (PyObject *)np; - oldbytes = Py_SIZE(a) * a->ob_descr->itemsize; - newbytes = oldbytes * n; - /* this follows the code in unicode_repeat */ - if (oldbytes == 1) { - memset(np->ob_item, a->ob_item[0], newbytes); - } else { - Py_ssize_t done = oldbytes; - memcpy(np->ob_item, a->ob_item, oldbytes); - while (done < newbytes) { - Py_ssize_t ncopy = (done <= newbytes-done) ? done : newbytes-done; - memcpy(np->ob_item+done, np->ob_item, ncopy); - done += ncopy; - } - } + + const Py_ssize_t oldbytes = array_length * a->ob_descr->itemsize; + const Py_ssize_t newbytes = oldbytes * n; + _PyBytes_Repeat(np->ob_item, newbytes, a->ob_item, oldbytes); + return (PyObject *)np; } @@ -1075,27 +1066,23 @@ array_inplace_concat(arrayobject *self, PyObject *bb) static PyObject * array_inplace_repeat(arrayobject *self, Py_ssize_t n) { - char *items, *p; - Py_ssize_t size, i; + const Py_ssize_t array_size = Py_SIZE(self); - if (Py_SIZE(self) > 0) { + if (array_size > 0 && n != 1 ) { if (n < 0) n = 0; if ((self->ob_descr->itemsize != 0) && - (Py_SIZE(self) > PY_SSIZE_T_MAX / self->ob_descr->itemsize)) { + (array_size > PY_SSIZE_T_MAX / self->ob_descr->itemsize)) { return PyErr_NoMemory(); } - size = Py_SIZE(self) * self->ob_descr->itemsize; + Py_ssize_t size = array_size * self->ob_descr->itemsize; if (n > 0 && size > PY_SSIZE_T_MAX / n) { return PyErr_NoMemory(); } - if (array_resize(self, n * Py_SIZE(self)) == -1) + if (array_resize(self, n * array_size) == -1) return NULL; - items = p = self->ob_item; - for (i = 1; i < n; i++) { - p += size; - memcpy(p, items, size); - } + + _PyBytes_Repeat(self->ob_item, n*size, self->ob_item, size); } Py_INCREF(self); return (PyObject *)self; diff --git a/Objects/bytearrayobject.c b/Objects/bytearrayobject.c index 74267cffcc9a3..3849337fbf77f 100644 --- a/Objects/bytearrayobject.c +++ b/Objects/bytearrayobject.c @@ -4,6 +4,7 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_bytes_methods.h" +#include "pycore_bytesobject.h" #include "pycore_object.h" // _PyObject_GC_UNTRACK() #include "pycore_strhex.h" // _Py_strhex_with_sep() #include "pycore_long.h" // _PyLong_FromUnsignedChar() @@ -319,37 +320,16 @@ bytearray_iconcat(PyByteArrayObject *self, PyObject *other) static PyObject * bytearray_repeat(PyByteArrayObject *self, Py_ssize_t count) { - PyByteArrayObject *result; - Py_ssize_t mysize; - Py_ssize_t size; - const char *buf; - if (count < 0) count = 0; - mysize = Py_SIZE(self); + const Py_ssize_t mysize = Py_SIZE(self); if (count > 0 && mysize > PY_SSIZE_T_MAX / count) return PyErr_NoMemory(); - size = mysize * count; - result = (PyByteArrayObject *)PyByteArray_FromStringAndSize(NULL, size); - buf = PyByteArray_AS_STRING(self); + Py_ssize_t size = mysize * count; + PyByteArrayObject* result = (PyByteArrayObject *)PyByteArray_FromStringAndSize(NULL, size); + const char* buf = PyByteArray_AS_STRING(self); if (result != NULL && size != 0) { - if (mysize == 1) - memset(result->ob_bytes, buf[0], size); - else { - Py_ssize_t i, j; - - i = 0; - if (i < size) { - memcpy(result->ob_bytes, buf, mysize); - i = mysize; - } - // repeatedly double the number of bytes copied - while (i < size) { - j = Py_MIN(i, size - i); - memcpy(result->ob_bytes + i, result->ob_bytes, j); - i += j; - } - } + _PyBytes_Repeat(result->ob_bytes, size, buf, mysize); } return (PyObject *)result; } @@ -357,33 +337,22 @@ bytearray_repeat(PyByteArrayObject *self, Py_ssize_t count) static PyObject * bytearray_irepeat(PyByteArrayObject *self, Py_ssize_t count) { - Py_ssize_t mysize; - Py_ssize_t size; - char *buf; - if (count < 0) count = 0; - mysize = Py_SIZE(self); + else if (count == 1) { + Py_INCREF(self); + return (PyObject*)self; + } + + const Py_ssize_t mysize = Py_SIZE(self); if (count > 0 && mysize > PY_SSIZE_T_MAX / count) return PyErr_NoMemory(); - size = mysize * count; + const Py_ssize_t size = mysize * count; if (PyByteArray_Resize((PyObject *)self, size) < 0) return NULL; - buf = PyByteArray_AS_STRING(self); - if (mysize == 1) - memset(buf, buf[0], size); - else { - Py_ssize_t i, j; - - i = mysize; - // repeatedly double the number of bytes copied - while (i < size) { - j = Py_MIN(i, size - i); - memcpy(buf + i, buf, j); - i += j; - } - } + char* buf = PyByteArray_AS_STRING(self); + _PyBytes_Repeat(buf, size, buf, mysize); Py_INCREF(self); return (PyObject *)self; diff --git a/Objects/bytesobject.c b/Objects/bytesobject.c index 56c941a44bfd5..78c42c2c54b5f 100644 --- a/Objects/bytesobject.c +++ b/Objects/bytesobject.c @@ -4,7 +4,7 @@ #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() -#include "pycore_bytesobject.h" // _PyBytes_Find() +#include "pycore_bytesobject.h" // _PyBytes_Find(), _PyBytes_Repeat() #include "pycore_bytes_methods.h" // _Py_bytes_startswith() #include "pycore_call.h" // _PyObject_CallNoArgs() #include "pycore_format.h" // F_LJUST @@ -1421,8 +1421,6 @@ bytes_concat(PyObject *a, PyObject *b) static PyObject * bytes_repeat(PyBytesObject *a, Py_ssize_t n) { - Py_ssize_t i; - Py_ssize_t j; Py_ssize_t size; PyBytesObject *op; size_t nbytes; @@ -1457,20 +1455,9 @@ _Py_COMP_DIAG_IGNORE_DEPR_DECLS op->ob_shash = -1; _Py_COMP_DIAG_POP op->ob_sval[size] = '\0'; - if (Py_SIZE(a) == 1 && n > 0) { - memset(op->ob_sval, a->ob_sval[0] , n); - return (PyObject *) op; - } - i = 0; - if (i < size) { - memcpy(op->ob_sval, a->ob_sval, Py_SIZE(a)); - i = Py_SIZE(a); - } - while (i < size) { - j = (i <= size-i) ? i : size-i; - memcpy(op->ob_sval+i, op->ob_sval, j); - i += j; - } + + _PyBytes_Repeat(op->ob_sval, size, a->ob_sval, Py_SIZE(a)); + return (PyObject *) op; } @@ -3528,3 +3515,28 @@ _PyBytesWriter_WriteBytes(_PyBytesWriter *writer, void *ptr, return str; } + + +void +_PyBytes_Repeat(char* dest, Py_ssize_t len_dest, + const char* src, Py_ssize_t len_src) +{ + if (len_dest == 0) { + return; + } + if (len_src == 1) { + memset(dest, src[0], len_dest); + } + else { + if (src != dest) { + memcpy(dest, src, len_src); + } + Py_ssize_t copied = len_src; + while (copied < len_dest) { + Py_ssize_t bytes_to_copy = Py_MIN(copied, len_dest - copied); + memcpy(dest + copied, dest, bytes_to_copy); + copied += bytes_to_copy; + } + } +} + diff --git a/Objects/unicodeobject.c b/Objects/unicodeobject.c index ce3ebce1ff72d..72f9245afb79a 100644 --- a/Objects/unicodeobject.c +++ b/Objects/unicodeobject.c @@ -42,6 +42,7 @@ OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. #include "Python.h" #include "pycore_abstract.h" // _PyIndex_Check() #include "pycore_atomic_funcs.h" // _Py_atomic_size_get() +#include "pycore_bytesobject.h" // _PyBytes_Repeat() #include "pycore_bytes_methods.h" // _Py_bytes_lower() #include "pycore_format.h" // F_LJUST #include "pycore_initconfig.h" // _PyStatus_OK() @@ -12782,17 +12783,10 @@ unicode_repeat(PyObject *str, Py_ssize_t len) } } else { - /* number of characters copied this far */ - Py_ssize_t done = PyUnicode_GET_LENGTH(str); Py_ssize_t char_size = PyUnicode_KIND(str); char *to = (char *) PyUnicode_DATA(u); - memcpy(to, PyUnicode_DATA(str), - PyUnicode_GET_LENGTH(str) * char_size); - while (done < nchars) { - n = (done <= nchars-done) ? done : nchars-done; - memcpy(to + (done * char_size), to, n * char_size); - done += n; - } + _PyBytes_Repeat(to, nchars * char_size, PyUnicode_DATA(str), + PyUnicode_GET_LENGTH(str) * char_size); } assert(_PyUnicode_CheckConsistency(u, 1));

1 0

bpo-28516: document contextlib.ExitStack.__enter__ behavior (GH-31636)
by JelleZijlstra March 28, 2022

March 28, 2022

https://github.com/python/cpython/commit/86384cf83f96fcaec03e2ad6516e2e24f2… commit: 86384cf83f96fcaec03e2ad6516e2e24f20d3b92 branch: main author: vidhya <96202776+Vidhyavinu(a)users.noreply.github.com> committer: JelleZijlstra <jelle.zijlstra(a)gmail.com> date: 2022-03-27T21:31:32-07:00 summary: bpo-28516: document contextlib.ExitStack.__enter__ behavior (GH-31636) The enter_context is updated with following information: 'The :meth:`__enter__` method returns the ExitStack instance, and performs no additional operations.' Co-authored-by: Jelle Zijlstra <jelle.zijlstra(a)gmail.com> files: M Doc/library/contextlib.rst diff --git a/Doc/library/contextlib.rst b/Doc/library/contextlib.rst index bb93088b3429e..38134c1f14319 100644 --- a/Doc/library/contextlib.rst +++ b/Doc/library/contextlib.rst @@ -502,6 +502,9 @@ Functions and classes provided: # the with statement, even if attempts to open files later # in the list raise an exception + The :meth:`__enter__` method returns the :class:`ExitStack` instance, and + performs no additional operations. + Each instance maintains a stack of registered callbacks that are called in reverse order when the instance is closed (either explicitly or implicitly at the end of a :keyword:`with` statement). Note that callbacks are *not*

1 0

ctypes docs: Fix array-length reference to "non-negative" from "positive" (GH-32097)
by JelleZijlstra March 28, 2022

March 28, 2022

https://github.com/python/cpython/commit/76f14b0463dc2c53911eaf95e85374e511… commit: 76f14b0463dc2c53911eaf95e85374e511ba9bcc branch: main author: Yonatan Goldschmidt <yon.goldschmidt(a)gmail.com> committer: JelleZijlstra <jelle.zijlstra(a)gmail.com> date: 2022-03-27T20:12:21-07:00 summary: ctypes docs: Fix array-length reference to "non-negative" from "positive" (GH-32097) files: M Doc/library/ctypes.rst diff --git a/Doc/library/ctypes.rst b/Doc/library/ctypes.rst index 6e147fc66eb14..dca4c74bab771 100644 --- a/Doc/library/ctypes.rst +++ b/Doc/library/ctypes.rst @@ -2512,7 +2512,7 @@ Arrays and pointers Abstract base class for arrays. The recommended way to create concrete array types is by multiplying any - :mod:`ctypes` data type with a positive integer. Alternatively, you can subclass + :mod:`ctypes` data type with a non-negative integer. Alternatively, you can subclass this type and define :attr:`_length_` and :attr:`_type_` class variables. Array elements can be read and written using standard subscript and slice accesses; for slice reads, the resulting object is

1 0

bpo-44493: Add missing terminated NUL in sockaddr_un's length (GH-26866)
by gpshead March 27, 2022

March 27, 2022

https://github.com/python/cpython/commit/f6b3a07b7df60dc04d0260169ffef6e979… commit: f6b3a07b7df60dc04d0260169ffef6e9796a2124 branch: main author: ty <zonyitoo(a)users.noreply.github.com> committer: gpshead <greg(a)krypto.org> date: 2022-03-27T13:22:22-07:00 summary: bpo-44493: Add missing terminated NUL in sockaddr_un's length (GH-26866) Add missing terminated NUL in sockaddr_un's length - Linux: https://man7.org/linux/man-pages/man7/unix.7.html - *BSD: SUN_LEN files: A Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst M Modules/socketmodule.c diff --git a/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst b/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst new file mode 100644 index 0000000000000..390a7222bbf55 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-07-26-10-46-49.bpo-44493.xp3CRH.rst @@ -0,0 +1,3 @@ +Add missing terminated NUL in sockaddr_un's length + +This was potentially observable when using non-abstract AF_UNIX datagram sockets to processes written in another programming language. diff --git a/Modules/socketmodule.c b/Modules/socketmodule.c index 7f7af1895bd9f..c7bc10b5dbbbc 100644 --- a/Modules/socketmodule.c +++ b/Modules/socketmodule.c @@ -1689,6 +1689,8 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, "AF_UNIX path too long"); goto unix_out; } + + *len_ret = path.len + offsetof(struct sockaddr_un, sun_path); } else #endif /* linux */ @@ -1700,10 +1702,13 @@ getsockaddrarg(PySocketSockObject *s, PyObject *args, goto unix_out; } addr->sun_path[path.len] = 0; + + /* including the tailing NUL */ + *len_ret = path.len + offsetof(struct sockaddr_un, sun_path) + 1; } addr->sun_family = s->sock_family; memcpy(addr->sun_path, path.buf, path.len); - *len_ret = path.len + offsetof(struct sockaddr_un, sun_path); + retval = 1; unix_out: PyBuffer_Release(&path);

1 0

bpo-47127: Specialize calls for fastcall c methods with keywords (GH-32125)
by Fidget-Spinner March 27, 2022

March 27, 2022

https://github.com/python/cpython/commit/58448cbd96f77ebc6fca1f8030bedc7744… commit: 58448cbd96f77ebc6fca1f8030bedc7744eb66ef branch: main author: Kumar Aditya <59607654+kumaraditya303(a)users.noreply.github.com> committer: Fidget-Spinner <kenjin4096(a)gmail.com> date: 2022-03-28T03:53:25+08:00 summary: bpo-47127: Specialize calls for fastcall c methods with keywords (GH-32125) * add PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS files: A Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst M Include/opcode.h M Lib/opcode.py M Python/ceval.c M Python/opcode_targets.h M Python/specialize.c diff --git a/Include/opcode.h b/Include/opcode.h index dfc7b72e3cdc6..0ce7c158bbd58 100644 --- a/Include/opcode.h +++ b/Include/opcode.h @@ -157,32 +157,33 @@ extern "C" { #define PRECALL_BOUND_METHOD 62 #define PRECALL_BUILTIN_CLASS 63 #define PRECALL_BUILTIN_FAST_WITH_KEYWORDS 64 -#define PRECALL_NO_KW_BUILTIN_FAST 65 -#define PRECALL_NO_KW_BUILTIN_O 66 -#define PRECALL_NO_KW_ISINSTANCE 67 -#define PRECALL_NO_KW_LEN 72 -#define PRECALL_NO_KW_LIST_APPEND 73 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 76 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 77 -#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 78 -#define PRECALL_NO_KW_STR_1 79 -#define PRECALL_NO_KW_TUPLE_1 80 -#define PRECALL_NO_KW_TYPE_1 81 -#define PRECALL_PYFUNC 140 -#define RESUME_QUICK 141 -#define STORE_ATTR_ADAPTIVE 143 -#define STORE_ATTR_INSTANCE_VALUE 150 -#define STORE_ATTR_SLOT 153 -#define STORE_ATTR_WITH_HINT 154 -#define STORE_FAST__LOAD_FAST 158 -#define STORE_FAST__STORE_FAST 159 -#define STORE_SUBSCR_ADAPTIVE 161 -#define STORE_SUBSCR_DICT 167 -#define STORE_SUBSCR_LIST_INT 168 -#define UNPACK_SEQUENCE_ADAPTIVE 169 -#define UNPACK_SEQUENCE_LIST 170 -#define UNPACK_SEQUENCE_TUPLE 173 -#define UNPACK_SEQUENCE_TWO_TUPLE 174 +#define PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS 65 +#define PRECALL_NO_KW_BUILTIN_FAST 66 +#define PRECALL_NO_KW_BUILTIN_O 67 +#define PRECALL_NO_KW_ISINSTANCE 72 +#define PRECALL_NO_KW_LEN 73 +#define PRECALL_NO_KW_LIST_APPEND 76 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST 77 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS 78 +#define PRECALL_NO_KW_METHOD_DESCRIPTOR_O 79 +#define PRECALL_NO_KW_STR_1 80 +#define PRECALL_NO_KW_TUPLE_1 81 +#define PRECALL_NO_KW_TYPE_1 140 +#define PRECALL_PYFUNC 141 +#define RESUME_QUICK 143 +#define STORE_ATTR_ADAPTIVE 150 +#define STORE_ATTR_INSTANCE_VALUE 153 +#define STORE_ATTR_SLOT 154 +#define STORE_ATTR_WITH_HINT 158 +#define STORE_FAST__LOAD_FAST 159 +#define STORE_FAST__STORE_FAST 161 +#define STORE_SUBSCR_ADAPTIVE 167 +#define STORE_SUBSCR_DICT 168 +#define STORE_SUBSCR_LIST_INT 169 +#define UNPACK_SEQUENCE_ADAPTIVE 170 +#define UNPACK_SEQUENCE_LIST 173 +#define UNPACK_SEQUENCE_TUPLE 174 +#define UNPACK_SEQUENCE_TWO_TUPLE 175 #define DO_TRACING 255 extern const uint8_t _PyOpcode_Caches[256]; @@ -347,6 +348,7 @@ const uint8_t _PyOpcode_Deopt[256] = { [PRECALL_BOUND_METHOD] = PRECALL, [PRECALL_BUILTIN_CLASS] = PRECALL, [PRECALL_BUILTIN_FAST_WITH_KEYWORDS] = PRECALL, + [PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS] = PRECALL, [PRECALL_NO_KW_BUILTIN_FAST] = PRECALL, [PRECALL_NO_KW_BUILTIN_O] = PRECALL, [PRECALL_NO_KW_ISINSTANCE] = PRECALL, diff --git a/Lib/opcode.py b/Lib/opcode.py index 7a52c13579af7..0b463d3f183aa 100644 --- a/Lib/opcode.py +++ b/Lib/opcode.py @@ -294,6 +294,7 @@ def jabs_op(name, op, entries=0): "PRECALL_BOUND_METHOD", "PRECALL_BUILTIN_CLASS", "PRECALL_BUILTIN_FAST_WITH_KEYWORDS", + "PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS", "PRECALL_NO_KW_BUILTIN_FAST", "PRECALL_NO_KW_BUILTIN_O", "PRECALL_NO_KW_ISINSTANCE", diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst b/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst new file mode 100644 index 0000000000000..c4ec9774429d4 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-03-26-12-21-53.bpo-47127.Mh86RB.rst @@ -0,0 +1 @@ +Speed up calls to c functions with keyword arguments by 25% with specialization. Patch by Kumar Aditya. diff --git a/Python/ceval.c b/Python/ceval.c index 4824b192f4e48..8b59f4233a7b2 100644 --- a/Python/ceval.c +++ b/Python/ceval.c @@ -5090,6 +5090,38 @@ _PyEval_EvalFrameDefault(PyThreadState *tstate, _PyInterpreterFrame *frame, int DISPATCH(); } + TARGET(PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS) { + int is_meth = is_method(stack_pointer, oparg); + int total_args = oparg + is_meth; + PyObject *callable = PEEK(total_args + 1); + DEOPT_IF(!Py_IS_TYPE(callable, &PyMethodDescr_Type), PRECALL); + PyMethodDef *meth = ((PyMethodDescrObject *)callable)->d_method; + DEOPT_IF(meth->ml_flags != (METH_FASTCALL|METH_KEYWORDS), PRECALL); + STAT_INC(PRECALL, hit); + SKIP_CALL(); + int nargs = total_args-1; + STACK_SHRINK(nargs); + _PyCFunctionFastWithKeywords cfunc = (_PyCFunctionFastWithKeywords)(void(*)(void))meth->ml_meth; + PyObject *self = TOP(); + PyObject *res = cfunc(self, stack_pointer, nargs - KWNAMES_LEN(), call_shape.kwnames); + assert((res != NULL) ^ (_PyErr_Occurred(tstate) != NULL)); + call_shape.kwnames = NULL; + + /* Free the arguments. */ + for (int i = 0; i < nargs; i++) { + Py_DECREF(stack_pointer[i]); + } + Py_DECREF(self); + STACK_SHRINK(2-is_meth); + SET_TOP(res); + Py_DECREF(callable); + if (res == NULL) { + goto error; + } + CHECK_EVAL_BREAKER(); + DISPATCH(); + } + TARGET(PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS) { assert(call_shape.kwnames == NULL); assert(oparg == 0 || oparg == 1); diff --git a/Python/opcode_targets.h b/Python/opcode_targets.h index 2aa6471abf99a..dbcacee7e0205 100644 --- a/Python/opcode_targets.h +++ b/Python/opcode_targets.h @@ -64,23 +64,23 @@ static void *opcode_targets[256] = { &&TARGET_PRECALL_BOUND_METHOD, &&TARGET_PRECALL_BUILTIN_CLASS, &&TARGET_PRECALL_BUILTIN_FAST_WITH_KEYWORDS, + &&TARGET_PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS, &&TARGET_PRECALL_NO_KW_BUILTIN_FAST, &&TARGET_PRECALL_NO_KW_BUILTIN_O, - &&TARGET_PRECALL_NO_KW_ISINSTANCE, &&TARGET_GET_ITER, &&TARGET_GET_YIELD_FROM_ITER, &&TARGET_PRINT_EXPR, &&TARGET_LOAD_BUILD_CLASS, + &&TARGET_PRECALL_NO_KW_ISINSTANCE, &&TARGET_PRECALL_NO_KW_LEN, - &&TARGET_PRECALL_NO_KW_LIST_APPEND, &&TARGET_LOAD_ASSERTION_ERROR, &&TARGET_RETURN_GENERATOR, + &&TARGET_PRECALL_NO_KW_LIST_APPEND, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_NOARGS, &&TARGET_PRECALL_NO_KW_METHOD_DESCRIPTOR_O, &&TARGET_PRECALL_NO_KW_STR_1, &&TARGET_PRECALL_NO_KW_TUPLE_1, - &&TARGET_PRECALL_NO_KW_TYPE_1, &&TARGET_LIST_TO_TUPLE, &&TARGET_RETURN_VALUE, &&TARGET_IMPORT_STAR, @@ -139,39 +139,40 @@ static void *opcode_targets[256] = { &&TARGET_LOAD_DEREF, &&TARGET_STORE_DEREF, &&TARGET_DELETE_DEREF, + &&TARGET_PRECALL_NO_KW_TYPE_1, &&TARGET_PRECALL_PYFUNC, - &&TARGET_RESUME_QUICK, &&TARGET_CALL_FUNCTION_EX, - &&TARGET_STORE_ATTR_ADAPTIVE, + &&TARGET_RESUME_QUICK, &&TARGET_EXTENDED_ARG, &&TARGET_LIST_APPEND, &&TARGET_SET_ADD, &&TARGET_MAP_ADD, &&TARGET_LOAD_CLASSDEREF, &&TARGET_COPY_FREE_VARS, - &&TARGET_STORE_ATTR_INSTANCE_VALUE, + &&TARGET_STORE_ATTR_ADAPTIVE, &&TARGET_RESUME, &&TARGET_MATCH_CLASS, + &&TARGET_STORE_ATTR_INSTANCE_VALUE, &&TARGET_STORE_ATTR_SLOT, - &&TARGET_STORE_ATTR_WITH_HINT, &&TARGET_FORMAT_VALUE, &&TARGET_BUILD_CONST_KEY_MAP, &&TARGET_BUILD_STRING, + &&TARGET_STORE_ATTR_WITH_HINT, &&TARGET_STORE_FAST__LOAD_FAST, - &&TARGET_STORE_FAST__STORE_FAST, &&TARGET_LOAD_METHOD, - &&TARGET_STORE_SUBSCR_ADAPTIVE, + &&TARGET_STORE_FAST__STORE_FAST, &&TARGET_LIST_EXTEND, &&TARGET_SET_UPDATE, &&TARGET_DICT_MERGE, &&TARGET_DICT_UPDATE, &&TARGET_PRECALL, + &&TARGET_STORE_SUBSCR_ADAPTIVE, &&TARGET_STORE_SUBSCR_DICT, &&TARGET_STORE_SUBSCR_LIST_INT, &&TARGET_UNPACK_SEQUENCE_ADAPTIVE, - &&TARGET_UNPACK_SEQUENCE_LIST, &&TARGET_CALL, &&TARGET_KW_NAMES, + &&TARGET_UNPACK_SEQUENCE_LIST, &&TARGET_UNPACK_SEQUENCE_TUPLE, &&TARGET_UNPACK_SEQUENCE_TWO_TUPLE, &&_unknown_opcode, @@ -253,6 +254,5 @@ static void *opcode_targets[256] = { &&_unknown_opcode, &&_unknown_opcode, &&_unknown_opcode, - &&_unknown_opcode, &&TARGET_DO_TRACING }; diff --git a/Python/specialize.c b/Python/specialize.c index 720bc7f241586..5839d7629466d 100644 --- a/Python/specialize.c +++ b/Python/specialize.c @@ -1446,6 +1446,10 @@ specialize_method_descriptor(PyMethodDescrObject *descr, _Py_CODEUNIT *instr, _Py_SET_OPCODE(*instr, PRECALL_NO_KW_METHOD_DESCRIPTOR_FAST); return 0; } + case METH_FASTCALL|METH_KEYWORDS: { + _Py_SET_OPCODE(*instr, PRECALL_METHOD_DESCRIPTOR_FAST_WITH_KEYWORDS); + return 0; + } } SPECIALIZATION_FAIL(PRECALL, builtin_call_fail_kind(descr->d_method->ml_flags)); return -1;

1 0

bpo-46429: tweak deepfreeze output (#32107)
by gvanrossum March 27, 2022

March 27, 2022

https://github.com/python/cpython/commit/785cc6770588de087d09e89a69110af254… commit: 785cc6770588de087d09e89a69110af2542be208 branch: main author: Kumar Aditya <59607654+kumaraditya303(a)users.noreply.github.com> committer: gvanrossum <gvanrossum(a)gmail.com> date: 2022-03-27T11:46:22-07:00 summary: bpo-46429: tweak deepfreeze output (#32107) files: M Tools/scripts/deepfreeze.py diff --git a/Tools/scripts/deepfreeze.py b/Tools/scripts/deepfreeze.py index 1831c15784af7..dfa4b3a8eeb01 100644 --- a/Tools/scripts/deepfreeze.py +++ b/Tools/scripts/deepfreeze.py @@ -170,6 +170,8 @@ def generate_bytes(self, name: str, b: bytes) -> str: def generate_unicode(self, name: str, s: str) -> str: if s in identifiers: return f"&_Py_ID({s})" + if re.match(r'\A[A-Za-z0-9_]+\Z', s): + name = f"const_str_{s}" kind, ascii = analyze_character_width(s) if kind == PyUnicode_1BYTE_KIND: datatype = "uint8_t" @@ -326,6 +328,10 @@ def _generate_int_for_bits(self, name: str, i: int, digit: int) -> None: def generate_int(self, name: str, i: int) -> str: if -5 <= i <= 256: return f"(PyObject *)&_PyLong_SMALL_INTS[_PY_NSMALLNEGINTS + {i}]" + if i >= 0: + name = f"const_int_{i}" + else: + name = f"const_int_negative_{abs(i)}" if abs(i) < 2**15: self._generate_int_for_bits(name, i, 2**15) else:

1 0

bpo-45171: Remove tests of deprecated logger.warn(). (GH-32139)
by vsajip March 27, 2022

March 27, 2022

https://github.com/python/cpython/commit/c12ba6b2ff7908c8970b978f149d51ecd3… commit: c12ba6b2ff7908c8970b978f149d51ecd3fb195c branch: main author: Jouke Witteveen <j.witteveen(a)gmail.com> committer: vsajip <vinay_sajip(a)yahoo.co.uk> date: 2022-03-27T19:22:05+01:00 summary: bpo-45171: Remove tests of deprecated logger.warn(). (GH-32139) files: M Lib/logging/__init__.py M Lib/test/test_logging.py diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index e8054fb1f72ac..d6315b047334e 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -1568,7 +1568,7 @@ def findCaller(self, stack_info=False, stacklevel=1): while stacklevel > 0: next_f = f.f_back if next_f is None: - ##TODO: We've got options here + ## We've got options here. ## If we want to use the last (deepest) frame: break ## If we want to mimic the warnings module: diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 00e354147a27c..f6f5977df2a1e 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5075,9 +5075,6 @@ def outer(): self.assertEqual(records[-1].funcName, 'outer') self.assertGreater(records[-1].lineno, lineno) lineno = records[-1].lineno - trigger = self.logger.warn - outer() - self.assertEqual(records[-1].funcName, 'outer') root_logger = logging.getLogger() root_logger.addHandler(self.recording) trigger = logging.warning

1 0

bpo-45171: Fix stacklevel handling in logging. (GH-28287)
by vsajip March 27, 2022

March 27, 2022

https://github.com/python/cpython/commit/5ca6d7469be53960843df39bb900e9c335… commit: 5ca6d7469be53960843df39bb900e9c3359f127f branch: main author: Jouke Witteveen <j.witteveen(a)gmail.com> committer: vsajip <vinay_sajip(a)yahoo.co.uk> date: 2022-03-27T14:49:28+01:00 summary: bpo-45171: Fix stacklevel handling in logging. (GH-28287) files: A Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst M Lib/logging/__init__.py M Lib/test/test_logging.py diff --git a/Lib/logging/__init__.py b/Lib/logging/__init__.py index 160b1afcee0f9..e8054fb1f72ac 100644 --- a/Lib/logging/__init__.py +++ b/Lib/logging/__init__.py @@ -159,8 +159,8 @@ def addLevelName(level, levelName): finally: _releaseLock() -if hasattr(sys, '_getframe'): - currentframe = lambda: sys._getframe(3) +if hasattr(sys, "_getframe"): + currentframe = lambda: sys._getframe(1) else: #pragma: no cover def currentframe(): """Return the frame object for the caller's stack frame.""" @@ -184,13 +184,18 @@ def currentframe(): _srcfile = os.path.normcase(addLevelName.__code__.co_filename) # _srcfile is only used in conjunction with sys._getframe(). -# To provide compatibility with older versions of Python, set _srcfile -# to None if _getframe() is not available; this value will prevent -# findCaller() from being called. You can also do this if you want to avoid -# the overhead of fetching caller information, even when _getframe() is -# available. -#if not hasattr(sys, '_getframe'): -# _srcfile = None +# Setting _srcfile to None will prevent findCaller() from being called. This +# way, you can avoid the overhead of fetching caller information. + +# The following is based on warnings._is_internal_frame. It makes sure that +# frames of the import mechanism are skipped when logging at module level and +# using a stacklevel value greater than one. +def _is_internal_frame(frame): + """Signal whether the frame is a CPython or logging module internal.""" + filename = os.path.normcase(frame.f_code.co_filename) + return filename == _srcfile or ( + "importlib" in filename and "_bootstrap" in filename + ) def _checkLevel(level): @@ -1558,33 +1563,31 @@ def findCaller(self, stack_info=False, stacklevel=1): f = currentframe() #On some versions of IronPython, currentframe() returns None if #IronPython isn't run with -X:Frames. - if f is not None: - f = f.f_back - orig_f = f - while f and stacklevel > 1: - f = f.f_back - stacklevel -= 1 - if not f: - f = orig_f - rv = "(unknown file)", 0, "(unknown function)", None - while hasattr(f, "f_code"): - co = f.f_code - filename = os.path.normcase(co.co_filename) - if filename == _srcfile: - f = f.f_back - continue - sinfo = None - if stack_info: - sio = io.StringIO() - sio.write('Stack (most recent call last):\n') + if f is None: + return "(unknown file)", 0, "(unknown function)", None + while stacklevel > 0: + next_f = f.f_back + if next_f is None: + ##TODO: We've got options here + ## If we want to use the last (deepest) frame: + break + ## If we want to mimic the warnings module: + #return ("sys", 1, "(unknown function)", None) + ## If we want to be pedantic: + #raise ValueError("call stack is not deep enough") + f = next_f + if not _is_internal_frame(f): + stacklevel -= 1 + co = f.f_code + sinfo = None + if stack_info: + with io.StringIO() as sio: + sio.write("Stack (most recent call last):\n") traceback.print_stack(f, file=sio) sinfo = sio.getvalue() if sinfo[-1] == '\n': sinfo = sinfo[:-1] - sio.close() - rv = (co.co_filename, f.f_lineno, co.co_name, sinfo) - break - return rv + return co.co_filename, f.f_lineno, co.co_name, sinfo def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None, sinfo=None): diff --git a/Lib/test/test_logging.py b/Lib/test/test_logging.py index 5f72a6dd5871a..00e354147a27c 100644 --- a/Lib/test/test_logging.py +++ b/Lib/test/test_logging.py @@ -5050,9 +5050,10 @@ def test_find_caller_with_stack_info(self): def test_find_caller_with_stacklevel(self): the_level = 1 + trigger = self.logger.warning def innermost(): - self.logger.warning('test', stacklevel=the_level) + trigger('test', stacklevel=the_level) def inner(): innermost() @@ -5074,6 +5075,16 @@ def outer(): self.assertEqual(records[-1].funcName, 'outer') self.assertGreater(records[-1].lineno, lineno) lineno = records[-1].lineno + trigger = self.logger.warn + outer() + self.assertEqual(records[-1].funcName, 'outer') + root_logger = logging.getLogger() + root_logger.addHandler(self.recording) + trigger = logging.warning + outer() + self.assertEqual(records[-1].funcName, 'outer') + root_logger.removeHandler(self.recording) + trigger = self.logger.warning the_level += 1 outer() self.assertEqual(records[-1].funcName, 'test_find_caller_with_stacklevel') diff --git a/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst b/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst new file mode 100644 index 0000000000000..eaa3fb2915e52 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2021-09-11-16-06-54.bpo-45171.ec597j.rst @@ -0,0 +1,4 @@ +Fix handling of the ``stacklevel`` argument to logging functions in the +:mod:`logging` module so that it is consistent accross all logging functions +and, as advertised, similar to the ``stacklevel`` argument used in +:meth:`~warnings.warn`.

1 0