Mailman 3 January 2020 - Python-checkins

bpo-39485: fix corner-case in method-detection of mock (GH-18252)
by Carl Friedrich Bolz-Tereick 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/a327677905956ae0b239ff430a1346dfe2… commit: a327677905956ae0b239ff430a1346dfe265709e branch: master author: Carl Friedrich Bolz-Tereick <cfbolz(a)gmx.de> committer: GitHub <noreply(a)github.com> date: 2020-01-29T15:43:37Z summary: bpo-39485: fix corner-case in method-detection of mock (GH-18252) Replace check for whether something is a method in the mock module. The previous version fails on PyPy, because there no method wrappers exist (everything looks like a regular Python-defined function). Thus the isinstance(getattr(result, '__get__', None), MethodWrapperTypes) check returns True for any descriptor, not just methods. This condition could also return erroneously True in CPython for C-defined descriptors. Instead to decide whether something is a method, just check directly whether it's a function defined on the class. This passes all tests on CPython and fixes the bug on PyPy. files: A Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst M Lib/unittest/mock.py diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index a3d8b6eab41a9..37ea5c7e45c8b 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -2748,7 +2748,7 @@ def _must_skip(spec, entry, is_type): continue if isinstance(result, (staticmethod, classmethod)): return False - elif isinstance(getattr(result, '__get__', None), MethodWrapperTypes): + elif isinstance(result, FunctionTypes): # Normal method => skip if looked up on type # (if looked up on instance, self is already skipped) return is_type @@ -2778,10 +2778,6 @@ def __init__(self, spec, spec_set=False, parent=None, type(ANY.__eq__), ) -MethodWrapperTypes = ( - type(ANY.__eq__.__get__), -) - file_spec = None diff --git a/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst b/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst new file mode 100644 index 0000000000000..f62c31fc686ad --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-01-29-14-58-27.bpo-39485.Zy3ot6.rst @@ -0,0 +1,3 @@ +Fix a bug in :func:`unittest.mock.create_autospec` that would complain about +the wrong number of arguments for custom descriptors defined in an extension +module returning functions.

1 0

bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247)
by Miss Islington (bot) 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/2b675f0c8fd96f61977f6dc636f44fbd55… commit: 2b675f0c8fd96f61977f6dc636f44fbd5587b6b3 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T06:41:49-08:00 summary: bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247) XFS filesystem is limited to 32-bit timestamp, but the utimensat() syscall doesn't fail. Moreover, there is a VFS bug which returns a cached timestamp which is different than the value on disk. https://bugzilla.redhat.com/show_bug.cgi?id=1795576 https://bugs.python.org/issue39460GH-msg360952 (cherry picked from commit 3cb49b62e61208efcefbc04414e769fc173f205d) Co-authored-by: Victor Stinner <vstinner(a)python.org> files: M Lib/test/test_zipfile.py diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index f03c044eae3a6..0737c343cbf5f 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -602,6 +602,18 @@ def test_add_file_after_2107(self): except OverflowError: self.skipTest('Host fs cannot set timestamp to required value.') + mtime_ns = os.stat(TESTFN).st_mtime_ns + if mtime_ns != (4386268800 * 10**9): + # XFS filesystem is limited to 32-bit timestamp, but the syscall + # didn't fail. Moreover, there is a VFS bug which returns + # a cached timestamp which is different than the value on disk. + # + # Test st_mtime_ns rather than st_mtime to avoid rounding issues. + # + # https://bugzilla.redhat.com/show_bug.cgi?id=1795576 + # https://bugs.python.org/issue39460#msg360952 + self.skipTest(f"Linux VFS/XFS kernel bug detected: {mtime_ns=}") + with zipfile.ZipFile(TESTFN2, "w") as zipfp: self.assertRaises(struct.error, zipfp.write, TESTFN)

1 0

bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247)
by Victor Stinner 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/3cb49b62e61208efcefbc04414e769fc17… commit: 3cb49b62e61208efcefbc04414e769fc173f205d branch: master author: Victor Stinner <vstinner(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-01-29T15:23:29+01:00 summary: bpo-39460: Fix test_zipfile.test_add_file_after_2107() (GH-18247) XFS filesystem is limited to 32-bit timestamp, but the utimensat() syscall doesn't fail. Moreover, there is a VFS bug which returns a cached timestamp which is different than the value on disk. https://bugzilla.redhat.com/show_bug.cgi?id=1795576 https://bugs.python.org/issue39460#msg360952 files: M Lib/test/test_zipfile.py diff --git a/Lib/test/test_zipfile.py b/Lib/test/test_zipfile.py index 66f05ac1f3aef..4c20bfd7e2cd7 100644 --- a/Lib/test/test_zipfile.py +++ b/Lib/test/test_zipfile.py @@ -616,6 +616,18 @@ def test_add_file_after_2107(self): except OverflowError: self.skipTest('Host fs cannot set timestamp to required value.') + mtime_ns = os.stat(TESTFN).st_mtime_ns + if mtime_ns != (4386268800 * 10**9): + # XFS filesystem is limited to 32-bit timestamp, but the syscall + # didn't fail. Moreover, there is a VFS bug which returns + # a cached timestamp which is different than the value on disk. + # + # Test st_mtime_ns rather than st_mtime to avoid rounding issues. + # + # https://bugzilla.redhat.com/show_bug.cgi?id=1795576 + # https://bugs.python.org/issue39460#msg360952 + self.skipTest(f"Linux VFS/XFS kernel bug detected: {mtime_ns=}") + with zipfile.ZipFile(TESTFN2, "w") as zipfp: self.assertRaises(struct.error, zipfp.write, TESTFN)

1 0

Doc: Fix external links to functional programming tutorial. (GH-18249)
by Miss Islington (bot) 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/d1fa90584f5f4bb0c47743d88a6520ba40… commit: d1fa90584f5f4bb0c47743d88a6520ba40e846a0 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T05:17:40-08:00 summary: Doc: Fix external links to functional programming tutorial. (GH-18249) (cherry picked from commit 35eac4500a8bd89b087407f59ba337343b22d403) Co-authored-by: Julien Palard <julien(a)palard.fr> files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index f8f2aac70f9b0..74e861480d2ff 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1229,9 +1229,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 <https://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, -`part 2 <https://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and -`part 3 <https://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, +`part 1 <https://developer.ibm.com/articles/l-prog/>`__, +`part 2 <https://developer.ibm.com/tutorials/l-prog2/>`__, and +`part 3 <https://developer.ibm.com/tutorials/l-prog3/>`__, Python documentation

1 0

Doc: Fix external links to functional programming tutorial. (GH-18249)
by Miss Islington (bot) 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/a6559b436dc5663828eca055be62683e0b… commit: a6559b436dc5663828eca055be62683e0b0cf5bb branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T05:16:57-08:00 summary: Doc: Fix external links to functional programming tutorial. (GH-18249) (cherry picked from commit 35eac4500a8bd89b087407f59ba337343b22d403) Co-authored-by: Julien Palard <julien(a)palard.fr> files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index f8f2aac70f9b0..74e861480d2ff 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1229,9 +1229,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 <https://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, -`part 2 <https://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and -`part 3 <https://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, +`part 1 <https://developer.ibm.com/articles/l-prog/>`__, +`part 2 <https://developer.ibm.com/tutorials/l-prog2/>`__, and +`part 3 <https://developer.ibm.com/tutorials/l-prog3/>`__, Python documentation

1 0

Doc: Fix external links to functional programming tutorial. (GH-18249)
by Julien Palard 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/35eac4500a8bd89b087407f59ba337343b… commit: 35eac4500a8bd89b087407f59ba337343b22d403 branch: master author: Julien Palard <julien(a)palard.fr> committer: GitHub <noreply(a)github.com> date: 2020-01-29T14:10:54+01:00 summary: Doc: Fix external links to functional programming tutorial. (GH-18249) files: M Doc/howto/functional.rst diff --git a/Doc/howto/functional.rst b/Doc/howto/functional.rst index f8f2aac70f9b0..74e861480d2ff 100644 --- a/Doc/howto/functional.rst +++ b/Doc/howto/functional.rst @@ -1229,9 +1229,9 @@ Text Processing". Mertz also wrote a 3-part series of articles on functional programming for IBM's DeveloperWorks site; see -`part 1 <https://www.ibm.com/developerworks/linux/library/l-prog/index.html>`__, -`part 2 <https://www.ibm.com/developerworks/linux/library/l-prog2/index.html>`__, and -`part 3 <https://www.ibm.com/developerworks/linux/library/l-prog3/index.html>`__, +`part 1 <https://developer.ibm.com/articles/l-prog/>`__, +`part 2 <https://developer.ibm.com/tutorials/l-prog2/>`__, and +`part 3 <https://developer.ibm.com/tutorials/l-prog3/>`__, Python documentation

1 0

bpo-39153: Clarify C API *SetItem refcounting semantics (GH-18220)
by Miss Islington (bot) 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/526523c19322169a7f7507d9da291053df… commit: 526523c19322169a7f7507d9da291053df979412 branch: 3.8 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T03:29:35-08:00 summary: bpo-39153: Clarify C API *SetItem refcounting semantics (GH-18220) Some of the *SetItem methods in the C API steal a reference to the given value. This annotates the better behaved ones to assure the reader that these are not the ones with the inconsistent behaviour. * 📜🤖 Added by blurb_it. * make docs consistent with signature Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot](a)users.noreply.github.com> (cherry picked from commit e1e80002e28e1055f399a20918c49d50d093709e) Co-authored-by: Joannah Nanjekye <33177550+nanjekyejoannah(a)users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst M Doc/c-api/dict.rst M Doc/c-api/mapping.rst M Doc/c-api/object.rst diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index e7922dc0c73f2..e48c11d336b8c 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -62,19 +62,20 @@ Dictionary Objects .. c:function:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val) - Insert *value* into the dictionary *p* with a key of *key*. *key* must be + Insert *val* into the dictionary *p* with a key of *key*. *key* must be :term:`hashable`; if it isn't, :exc:`TypeError` will be raised. Return - ``0`` on success or ``-1`` on failure. + ``0`` on success or ``-1`` on failure. This function *does not* steal a + reference to *val*. .. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val) .. index:: single: PyUnicode_FromString() - Insert *value* into the dictionary *p* using *key* as a key. *key* should + Insert *val* into the dictionary *p* using *key* as a key. *key* should be a :c:type:`const char\*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on - failure. + failure. This function *does not* steal a reference to *val*. .. c:function:: int PyDict_DelItem(PyObject *p, PyObject *key) diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 6a80b033b651e..682160d1475c1 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -37,7 +37,8 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and Map the string *key* to the value *v* in object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[key] = v``. - See also :c:func:`PyObject_SetItem`. + See also :c:func:`PyObject_SetItem`. This function *does not* steal a + reference to *v*. .. c:function:: int PyMapping_DelItem(PyObject *o, PyObject *key) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index 404a98fb417bb..db815ae288b86 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -503,7 +503,8 @@ Object Protocol Map the object *key* to the value *v*. Raise an exception and return ``-1`` on failure; return ``0`` on success. This is the - equivalent of the Python statement ``o[key] = v``. + equivalent of the Python statement ``o[key] = v``. This function *does + not* steal a reference to *v*. .. c:function:: int PyObject_DelItem(PyObject *o, PyObject *key) diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst new file mode 100644 index 0000000000000..95be00b4b777f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst @@ -0,0 +1,5 @@ +Clarify refcounting semantics for the following functions: +- PyObject_SetItem +- PyMapping_SetItemString +- PyDict_SetItem +- PyDict_SetItemString \ No newline at end of file

1 0

bpo-39153: Clarify C API *SetItem refcounting semantics (GH-18220)
by Joannah Nanjekye 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/e1e80002e28e1055f399a20918c49d50d0… commit: e1e80002e28e1055f399a20918c49d50d093709e branch: master author: Joannah Nanjekye <33177550+nanjekyejoannah(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T21:20:53+10:00 summary: bpo-39153: Clarify C API *SetItem refcounting semantics (GH-18220) Some of the *SetItem methods in the C API steal a reference to the given value. This annotates the better behaved ones to assure the reader that these are not the ones with the inconsistent behaviour. * 📜🤖 Added by blurb_it. * make docs consistent with signature Co-authored-by: blurb-it[bot] <43283697+blurb-it[bot](a)users.noreply.github.com> files: A Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst M Doc/c-api/dict.rst M Doc/c-api/mapping.rst M Doc/c-api/object.rst diff --git a/Doc/c-api/dict.rst b/Doc/c-api/dict.rst index e7922dc0c73f2..e48c11d336b8c 100644 --- a/Doc/c-api/dict.rst +++ b/Doc/c-api/dict.rst @@ -62,19 +62,20 @@ Dictionary Objects .. c:function:: int PyDict_SetItem(PyObject *p, PyObject *key, PyObject *val) - Insert *value* into the dictionary *p* with a key of *key*. *key* must be + Insert *val* into the dictionary *p* with a key of *key*. *key* must be :term:`hashable`; if it isn't, :exc:`TypeError` will be raised. Return - ``0`` on success or ``-1`` on failure. + ``0`` on success or ``-1`` on failure. This function *does not* steal a + reference to *val*. .. c:function:: int PyDict_SetItemString(PyObject *p, const char *key, PyObject *val) .. index:: single: PyUnicode_FromString() - Insert *value* into the dictionary *p* using *key* as a key. *key* should + Insert *val* into the dictionary *p* using *key* as a key. *key* should be a :c:type:`const char\*`. The key object is created using ``PyUnicode_FromString(key)``. Return ``0`` on success or ``-1`` on - failure. + failure. This function *does not* steal a reference to *val*. .. c:function:: int PyDict_DelItem(PyObject *p, PyObject *key) diff --git a/Doc/c-api/mapping.rst b/Doc/c-api/mapping.rst index 6a80b033b651e..682160d1475c1 100644 --- a/Doc/c-api/mapping.rst +++ b/Doc/c-api/mapping.rst @@ -37,7 +37,8 @@ See also :c:func:`PyObject_GetItem`, :c:func:`PyObject_SetItem` and Map the string *key* to the value *v* in object *o*. Returns ``-1`` on failure. This is the equivalent of the Python statement ``o[key] = v``. - See also :c:func:`PyObject_SetItem`. + See also :c:func:`PyObject_SetItem`. This function *does not* steal a + reference to *v*. .. c:function:: int PyMapping_DelItem(PyObject *o, PyObject *key) diff --git a/Doc/c-api/object.rst b/Doc/c-api/object.rst index ca9db1aee2f87..2905fbbd0f70c 100644 --- a/Doc/c-api/object.rst +++ b/Doc/c-api/object.rst @@ -331,7 +331,8 @@ Object Protocol Map the object *key* to the value *v*. Raise an exception and return ``-1`` on failure; return ``0`` on success. This is the - equivalent of the Python statement ``o[key] = v``. + equivalent of the Python statement ``o[key] = v``. This function *does + not* steal a reference to *v*. .. c:function:: int PyObject_DelItem(PyObject *o, PyObject *key) diff --git a/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst new file mode 100644 index 0000000000000..95be00b4b777f --- /dev/null +++ b/Misc/NEWS.d/next/Documentation/2020-01-27-22-24-51.bpo-39153.Pjl8jV.rst @@ -0,0 +1,5 @@ +Clarify refcounting semantics for the following functions: +- PyObject_SetItem +- PyMapping_SetItemString +- PyDict_SetItem +- PyDict_SetItemString \ No newline at end of file

1 0

bpo-36051: Drop GIL during large bytes.join() (GH-17757)
by Bruce Merry 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/d07d9f4c43bc85a77021bcc7d77643f8eb… commit: d07d9f4c43bc85a77021bcc7d77643f8ebb605cf branch: master author: Bruce Merry <bmerry(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-01-29T16:09:24+09:00 summary: bpo-36051: Drop GIL during large bytes.join() (GH-17757) Improve multi-threaded performance by dropping the GIL in the fast path of bytes.join. To avoid increasing overhead for small joins, it is only done if the output size exceeds a threshold. files: A Misc/NEWS.d/next/Core and Builtins/2019-12-30-15-56-07.bpo-36051.imaVlq.rst M Lib/test/test_bytes.py M Misc/ACKS M Objects/stringlib/join.h diff --git a/Lib/test/test_bytes.py b/Lib/test/test_bytes.py index ddcf367f38fc8..770e2c5592cc6 100644 --- a/Lib/test/test_bytes.py +++ b/Lib/test/test_bytes.py @@ -547,9 +547,13 @@ def test_join(self): self.assertEqual(dot_join([bytearray(b"ab"), b"cd"]), b"ab.:cd") self.assertEqual(dot_join([b"ab", bytearray(b"cd")]), b"ab.:cd") # Stress it with many items - seq = [b"abc"] * 1000 - expected = b"abc" + b".:abc" * 999 + seq = [b"abc"] * 100000 + expected = b"abc" + b".:abc" * 99999 self.assertEqual(dot_join(seq), expected) + # Stress test with empty separator + seq = [b"abc"] * 100000 + expected = b"abc" * 100000 + self.assertEqual(self.type2test(b"").join(seq), expected) self.assertRaises(TypeError, self.type2test(b" ").join, None) # Error handling and cleanup when some item in the middle of the # sequence has the wrong type. diff --git a/Misc/ACKS b/Misc/ACKS index 7e4b81bfdee62..f3e368078124d 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -1106,6 +1106,7 @@ Ezio Melotti Doug Mennella Dimitri Merejkowsky Brian Merrell +Bruce Merry Alexis Métaireau Luke Mewburn Carl Meyer diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-12-30-15-56-07.bpo-36051.imaVlq.rst b/Misc/NEWS.d/next/Core and Builtins/2019-12-30-15-56-07.bpo-36051.imaVlq.rst new file mode 100644 index 0000000000000..f9d449216ebed --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-12-30-15-56-07.bpo-36051.imaVlq.rst @@ -0,0 +1 @@ +Drop the GIL during large ``bytes.join`` operations. Patch by Bruce Merry. diff --git a/Objects/stringlib/join.h b/Objects/stringlib/join.h index 6f314e1524eb5..4d023ed1a851e 100644 --- a/Objects/stringlib/join.h +++ b/Objects/stringlib/join.h @@ -18,6 +18,9 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) Py_buffer *buffers = NULL; #define NB_STATIC_BUFFERS 10 Py_buffer static_buffers[NB_STATIC_BUFFERS]; +#define GIL_THRESHOLD 1048576 + int drop_gil = 1; + PyThreadState *save; seq = PySequence_Fast(iterable, "can only join an iterable"); if (seq == NULL) { @@ -65,12 +68,21 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) buffers[i].buf = PyBytes_AS_STRING(item); buffers[i].len = PyBytes_GET_SIZE(item); } - else if (PyObject_GetBuffer(item, &buffers[i], PyBUF_SIMPLE) != 0) { - PyErr_Format(PyExc_TypeError, - "sequence item %zd: expected a bytes-like object, " - "%.80s found", - i, Py_TYPE(item)->tp_name); - goto error; + else { + if (PyObject_GetBuffer(item, &buffers[i], PyBUF_SIMPLE) != 0) { + PyErr_Format(PyExc_TypeError, + "sequence item %zd: expected a bytes-like object, " + "%.80s found", + i, Py_TYPE(item)->tp_name); + goto error; + } + /* If the backing objects are mutable, then dropping the GIL + * opens up race conditions where another thread tries to modify + * the object which we hold a buffer on it. Such code has data + * races anyway, but this is a conservative approach that avoids + * changing the behaviour of that data race. + */ + drop_gil = 0; } nbufs = i + 1; /* for error cleanup */ itemlen = buffers[i].len; @@ -102,6 +114,12 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) /* Catenate everything. */ p = STRINGLIB_STR(res); + if (sz < GIL_THRESHOLD) { + drop_gil = 0; /* Benefits are likely outweighed by the overheads */ + } + if (drop_gil) { + save = PyEval_SaveThread(); + } if (!seplen) { /* fast path */ for (i = 0; i < nbufs; i++) { @@ -110,19 +128,23 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) memcpy(p, q, n); p += n; } - goto done; } - for (i = 0; i < nbufs; i++) { - Py_ssize_t n; - char *q; - if (i) { - memcpy(p, sepstr, seplen); - p += seplen; + else { + for (i = 0; i < nbufs; i++) { + Py_ssize_t n; + char *q; + if (i) { + memcpy(p, sepstr, seplen); + p += seplen; + } + n = buffers[i].len; + q = buffers[i].buf; + memcpy(p, q, n); + p += n; } - n = buffers[i].len; - q = buffers[i].buf; - memcpy(p, q, n); - p += n; + } + if (drop_gil) { + PyEval_RestoreThread(save); } goto done; @@ -138,3 +160,4 @@ STRINGLIB(bytes_join)(PyObject *sep, PyObject *iterable) } #undef NB_STATIC_BUFFERS +#undef GIL_THRESHOLD

1 0

bpo-39401: Avoid unsafe DLL load on Windows 7 and earlier (GH-18231)
by Steve Dower 29 Jan '20

29 Jan '20

https://github.com/python/cpython/commit/6a65eba44bfd82ccc8bed4b5c6dd663754… commit: 6a65eba44bfd82ccc8bed4b5c6dd6637549955d5 branch: master author: Steve Dower <steve.dower(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-01-29T13:46:33+11:00 summary: bpo-39401: Avoid unsafe DLL load on Windows 7 and earlier (GH-18231) As Windows 7 is not supported by Python 3.9, we just replace the dynamic load with a static import. Backports will have a different fix to ensure they continue to behave the same. files: A Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst M PC/getpathp.c M PCbuild/pythoncore.vcxproj diff --git a/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst b/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst new file mode 100644 index 0000000000000..78274acfcb743 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2020-01-28-20-54-09.bpo-39401.he7h_A.rst @@ -0,0 +1 @@ +Avoid unsafe DLL load at startup on Windows 7 and earlier. diff --git a/PC/getpathp.c b/PC/getpathp.c index 085caf195a992..3b65b35ce6146 100644 --- a/PC/getpathp.c +++ b/PC/getpathp.c @@ -91,6 +91,7 @@ #endif #include <windows.h> +#include <pathcch.h> #include <shlwapi.h> #ifdef HAVE_SYS_TYPES_H @@ -242,42 +243,14 @@ ismodule(wchar_t *filename, int update_filename) stuff as fits will be appended. */ -static int _PathCchCombineEx_Initialized = 0; -typedef HRESULT(__stdcall *PPathCchCombineEx) (PWSTR pszPathOut, size_t cchPathOut, - PCWSTR pszPathIn, PCWSTR pszMore, - unsigned long dwFlags); -static PPathCchCombineEx _PathCchCombineEx; - static void join(wchar_t *buffer, const wchar_t *stuff) { - if (_PathCchCombineEx_Initialized == 0) { - HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll"); - if (pathapi) { - _PathCchCombineEx = (PPathCchCombineEx)GetProcAddress(pathapi, "PathCchCombineEx"); - } - else { - _PathCchCombineEx = NULL; - } - _PathCchCombineEx_Initialized = 1; - } - - if (_PathCchCombineEx) { - if (FAILED(_PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) { - Py_FatalError("buffer overflow in getpathp.c's join()"); - } - } else { - if (!PathCombineW(buffer, buffer, stuff)) { - Py_FatalError("buffer overflow in getpathp.c's join()"); - } + if (FAILED(PathCchCombineEx(buffer, MAXPATHLEN+1, buffer, stuff, 0))) { + Py_FatalError("buffer overflow in getpathp.c's join()"); } } -static int _PathCchCanonicalizeEx_Initialized = 0; -typedef HRESULT(__stdcall *PPathCchCanonicalizeEx) (PWSTR pszPathOut, size_t cchPathOut, - PCWSTR pszPathIn, unsigned long dwFlags); -static PPathCchCanonicalizeEx _PathCchCanonicalizeEx; - /* Call PathCchCanonicalizeEx(path): remove navigation elements such as "." and ".." to produce a direct, well-formed path. */ static PyStatus @@ -287,26 +260,8 @@ canonicalize(wchar_t *buffer, const wchar_t *path) return _PyStatus_NO_MEMORY(); } - if (_PathCchCanonicalizeEx_Initialized == 0) { - HMODULE pathapi = LoadLibraryW(L"api-ms-win-core-path-l1-1-0.dll"); - if (pathapi) { - _PathCchCanonicalizeEx = (PPathCchCanonicalizeEx)GetProcAddress(pathapi, "PathCchCanonicalizeEx"); - } - else { - _PathCchCanonicalizeEx = NULL; - } - _PathCchCanonicalizeEx_Initialized = 1; - } - - if (_PathCchCanonicalizeEx) { - if (FAILED(_PathCchCanonicalizeEx(buffer, MAXPATHLEN + 1, path, 0))) { - return INIT_ERR_BUFFER_OVERFLOW(); - } - } - else { - if (!PathCanonicalizeW(buffer, path)) { - return INIT_ERR_BUFFER_OVERFLOW(); - } + if (FAILED(PathCchCanonicalizeEx(buffer, MAXPATHLEN + 1, path, 0))) { + return INIT_ERR_BUFFER_OVERFLOW(); } return _PyStatus_OK(); } diff --git a/PCbuild/pythoncore.vcxproj b/PCbuild/pythoncore.vcxproj index f5be8aa4051e8..cfab2fa4e189c 100644 --- a/PCbuild/pythoncore.vcxproj +++ b/PCbuild/pythoncore.vcxproj @@ -106,7 +106,7 @@ <PreprocessorDefinitions Condition="$(IncludeExternals)">_Py_HAVE_ZLIB;%(PreprocessorDefinitions)</PreprocessorDefinitions> </ClCompile> <Link> - <AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;%(AdditionalDependencies)</AdditionalDependencies> + <AdditionalDependencies>version.lib;shlwapi.lib;ws2_32.lib;pathcch.lib;%(AdditionalDependencies)</AdditionalDependencies> </Link> </ItemDefinitionGroup> <ItemGroup>

1 0