Mailman 3 September 2020 - Python-checkins

[3.8] [3.9] bpo-41654: Fix deallocator of MemoryError to account for subclasses (GH-22020) (GH-22046)
by Pablo Galindo 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/77f4000ae0d43a2685face80e7f14d4aba… commit: 77f4000ae0d43a2685face80e7f14d4aba053973 branch: 3.8 author: Pablo Galindo <Pablogsal(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-01T21:40:48+01:00 summary: [3.8] [3.9] bpo-41654: Fix deallocator of MemoryError to account for subclasses (GH-22020) (GH-22046) When allocating MemoryError classes, there is some logic to use pre-allocated instances in a freelist only if the type that is being allocated is not a subclass of MemoryError. Unfortunately in the destructor this logic is not present so the freelist is altered even with subclasses of MemoryError.. (cherry picked from commit 9b648a95ccb4c3b14f1e87158f5c9f5dbb2f62c0) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com>. (cherry picked from commit 87e91ae2e5f81e096c32839f211c68a749a4435a) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst M Lib/test/test_exceptions.py M Objects/exceptions.c diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 3a32253157369..457b46e01ce31 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -1,6 +1,7 @@ # Python test set -- part 5, built-in exceptions import copy +import gc import os import sys import unittest @@ -1297,6 +1298,35 @@ def g(): next(i) next(i) + def test_memory_error_subclasses(self): + # bpo-41654: MemoryError instances use a freelist of objects that are + # linked using the 'dict' attribute when they are inactive/dead. + # Subclasses of MemoryError should not participate in the freelist + # schema. This test creates a MemoryError object and keeps it alive + # (therefore advancing the freelist) and then it creates and destroys a + # subclass object. Finally, it checks that creating a new MemoryError + # succeeds, proving that the freelist is not corrupted. + + class TestException(MemoryError): + pass + + try: + raise MemoryError + except MemoryError as exc: + inst = exc + + try: + raise TestException + except Exception: + pass + + for _ in range(10): + try: + raise MemoryError + except MemoryError as exc: + pass + + gc_collect() class ImportErrorTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst new file mode 100644 index 0000000000000..e05c3133e1262 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst @@ -0,0 +1,2 @@ +Fix a crash that occurred when destroying subclasses of +:class:`MemoryError`. Patch by Pablo Galindo. diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 38723d93653c3..966983810cd1e 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -2268,8 +2268,12 @@ MemoryError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { PyBaseExceptionObject *self; - if (type != (PyTypeObject *) PyExc_MemoryError) + /* If this is a subclass of MemoryError, don't use the freelist + * and just return a fresh object */ + if (type != (PyTypeObject *) PyExc_MemoryError) { return BaseException_new(type, args, kwds); + } + if (memerrors_freelist == NULL) return BaseException_new(type, args, kwds); /* Fetch object from freelist and revive it */ @@ -2289,8 +2293,14 @@ MemoryError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void MemoryError_dealloc(PyBaseExceptionObject *self) { - _PyObject_GC_UNTRACK(self); BaseException_clear(self); + + if (Py_TYPE(self) != PyExc_MemoryError) { + return Py_TYPE(self)->tp_free((PyObject *)self); + } + + _PyObject_GC_UNTRACK(self); + if (memerrors_numfree >= MEMERRORS_SAVE) Py_TYPE(self)->tp_free((PyObject *)self); else {

1 0

bpo-41654: Fix deallocator of MemoryError to account for subclasses (GH-22020)
by Pablo Galindo 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/9b648a95ccb4c3b14f1e87158f5c9f5dbb… commit: 9b648a95ccb4c3b14f1e87158f5c9f5dbb2f62c0 branch: master author: Pablo Galindo <Pablogsal(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-01T19:39:46+01:00 summary: bpo-41654: Fix deallocator of MemoryError to account for subclasses (GH-22020) When allocating MemoryError classes, there is some logic to use pre-allocated instances in a freelist only if the type that is being allocated is not a subclass of MemoryError. Unfortunately in the destructor this logic is not present so the freelist is altered even with subclasses of MemoryError. files: A Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst M Lib/test/test_exceptions.py M Objects/exceptions.c diff --git a/Lib/test/test_exceptions.py b/Lib/test/test_exceptions.py index 2ffe8caa03f81..1ec446887770e 100644 --- a/Lib/test/test_exceptions.py +++ b/Lib/test/test_exceptions.py @@ -1,6 +1,7 @@ # Python test set -- part 5, built-in exceptions import copy +import gc import os import sys import unittest @@ -1330,6 +1331,36 @@ def test_assert_shadowing(self): del AssertionError self.fail('Expected exception') + def test_memory_error_subclasses(self): + # bpo-41654: MemoryError instances use a freelist of objects that are + # linked using the 'dict' attribute when they are inactive/dead. + # Subclasses of MemoryError should not participate in the freelist + # schema. This test creates a MemoryError object and keeps it alive + # (therefore advancing the freelist) and then it creates and destroys a + # subclass object. Finally, it checks that creating a new MemoryError + # succeeds, proving that the freelist is not corrupted. + + class TestException(MemoryError): + pass + + try: + raise MemoryError + except MemoryError as exc: + inst = exc + + try: + raise TestException + except Exception: + pass + + for _ in range(10): + try: + raise MemoryError + except MemoryError as exc: + pass + + gc_collect() + class ImportErrorTests(unittest.TestCase): diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst new file mode 100644 index 0000000000000..e05c3133e1262 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-30-20-38-33.bpo-41654.HtnhAM.rst @@ -0,0 +1,2 @@ +Fix a crash that occurred when destroying subclasses of +:class:`MemoryError`. Patch by Pablo Galindo. diff --git a/Objects/exceptions.c b/Objects/exceptions.c index 1195ba17922dd..b08cbdd6aed35 100644 --- a/Objects/exceptions.c +++ b/Objects/exceptions.c @@ -2286,8 +2286,11 @@ MemoryError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) { PyBaseExceptionObject *self; - if (type != (PyTypeObject *) PyExc_MemoryError) + /* If this is a subclass of MemoryError, don't use the freelist + * and just return a fresh object */ + if (type != (PyTypeObject *) PyExc_MemoryError) { return BaseException_new(type, args, kwds); + } struct _Py_exc_state *state = get_exc_state(); if (state->memerrors_freelist == NULL) { @@ -2313,9 +2316,16 @@ MemoryError_new(PyTypeObject *type, PyObject *args, PyObject *kwds) static void MemoryError_dealloc(PyBaseExceptionObject *self) { - _PyObject_GC_UNTRACK(self); BaseException_clear(self); + /* If this is a subclass of MemoryError, we don't need to + * do anything in the free-list*/ + if (!Py_IS_TYPE(self, (PyTypeObject *) PyExc_MemoryError)) { + return Py_TYPE(self)->tp_free((PyObject *)self); + } + + _PyObject_GC_UNTRACK(self); + struct _Py_exc_state *state = get_exc_state(); if (state->memerrors_numfree >= MEMERRORS_SAVE) { Py_TYPE(self)->tp_free((PyObject *)self);

1 0

bpo-41528: Use math module in turtle (GH-21837)
by Marek Madejski 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/6844b56176c41f0a0e25fcd4fef5463bcd… commit: 6844b56176c41f0a0e25fcd4fef5463bcdbc7d7c branch: master author: Marek Madejski <marekmadejski(a)yandex.com> committer: GitHub <noreply(a)github.com> date: 2020-09-01T12:42:41-04:00 summary: bpo-41528: Use math module in turtle (GH-21837) Use angle-related functions from math module instead of reinventing the wheel. files: A Misc/NEWS.d/next/Library/2020-08-12-07-43-31.bpo-41528.bu83oD.rst M Lib/turtle.py diff --git a/Lib/turtle.py b/Lib/turtle.py index ee67a351b54f1..92d4e5dda9c2d 100644 --- a/Lib/turtle.py +++ b/Lib/turtle.py @@ -263,12 +263,12 @@ def __sub__(self, other): def __neg__(self): return Vec2D(-self[0], -self[1]) def __abs__(self): - return (self[0]**2 + self[1]**2)**0.5 + return math.hypot(*self) def rotate(self, angle): """rotate self counterclockwise by angle """ perp = Vec2D(-self[1], self[0]) - angle = angle * math.pi / 180.0 + angle = math.radians(angle) c, s = math.cos(angle), math.sin(angle) return Vec2D(self[0]*c+perp[0]*s, self[1]*c+perp[1]*s) def __getnewargs__(self): @@ -1597,7 +1597,7 @@ def radians(self): >>> turtle.heading() 1.5707963267948966 """ - self._setDegreesPerAU(2*math.pi) + self._setDegreesPerAU(math.tau) def _go(self, distance): """move turtle forward by specified distance""" @@ -1888,7 +1888,7 @@ def towards(self, x, y=None): elif isinstance(x, TNavigator): pos = x._position x, y = pos - self._position - result = round(math.atan2(y, x)*180.0/math.pi, 10) % 360.0 + result = round(math.degrees(math.atan2(y, x)), 10) % 360.0 result /= self._degreesPerAU return (self._angleOffset + self._angleOrient*result) % self._fullcircle @@ -1903,7 +1903,7 @@ def heading(self): 67.0 """ x, y = self._orient - result = round(math.atan2(y, x)*180.0/math.pi, 10) % 360.0 + result = round(math.degrees(math.atan2(y, x)), 10) % 360.0 result /= self._degreesPerAU return (self._angleOffset + self._angleOrient*result) % self._fullcircle @@ -1976,7 +1976,7 @@ def circle(self, radius, extent = None, steps = None): steps = 1+int(min(11+abs(radius)/6.0, 59.0)*frac) w = 1.0 * extent / steps w2 = 0.5 * w - l = 2.0 * radius * math.sin(w2*math.pi/180.0*self._degreesPerAU) + l = 2.0 * radius * math.sin(math.radians(w2)*self._degreesPerAU) if radius < 0: l, w, w2 = -l, -w, -w2 tr = self._tracer() @@ -2861,7 +2861,7 @@ def settiltangle(self, angle): >>> turtle.fd(50) """ tilt = -angle * self._degreesPerAU * self._angleOrient - tilt = (tilt * math.pi / 180.0) % (2*math.pi) + tilt = math.radians(tilt) % math.tau self.pen(resizemode="user", tilt=tilt) def tiltangle(self, angle=None): @@ -2885,7 +2885,7 @@ def tiltangle(self, angle=None): >>> turtle.tiltangle() """ if angle is None: - tilt = -self._tilt * (180.0/math.pi) * self._angleOrient + tilt = -math.degrees(self._tilt) * self._angleOrient return (tilt / self._degreesPerAU) % self._fullcircle else: self.settiltangle(angle) @@ -2939,7 +2939,7 @@ def shapetransform(self, t11=None, t12=None, t21=None, t22=None): if t11 * t22 - t12 * t21 == 0: raise TurtleGraphicsError("Bad shape transform matrix: must not be singular") self._shapetrafo = (m11, m12, m21, m22) - alfa = math.atan2(-m21, m11) % (2 * math.pi) + alfa = math.atan2(-m21, m11) % math.tau sa, ca = math.sin(alfa), math.cos(alfa) a11, a12, a21, a22 = (ca*m11 - sa*m21, ca*m12 - sa*m22, sa*m11 + ca*m21, sa*m12 + ca*m22) diff --git a/Misc/NEWS.d/next/Library/2020-08-12-07-43-31.bpo-41528.bu83oD.rst b/Misc/NEWS.d/next/Library/2020-08-12-07-43-31.bpo-41528.bu83oD.rst new file mode 100644 index 0000000000000..a4ba57c2438ed --- /dev/null +++ b/Misc/NEWS.d/next/Library/2020-08-12-07-43-31.bpo-41528.bu83oD.rst @@ -0,0 +1 @@ +turtle uses math module functions to convert degrees to radians and vice versa and to calculate vector norm \ No newline at end of file

1 0

bpo-41617: Fix pycore_bitutils.h to support clang 3.0 (GH-22042)
by Victor Stinner 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/e6905e4c82cc05897dc1bf5ab2b5b94b2b… commit: e6905e4c82cc05897dc1bf5ab2b5b94b2b043a7f branch: master author: Victor Stinner <vstinner(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-09-01T18:25:14+02:00 summary: bpo-41617: Fix pycore_bitutils.h to support clang 3.0 (GH-22042) __builtin_bswap16() is not available in LLVM clang 3.0. files: A Misc/NEWS.d/next/Build/2020-08-24-18-34-01.bpo-41617.sKKXz7.rst M Include/internal/pycore_bitutils.h diff --git a/Include/internal/pycore_bitutils.h b/Include/internal/pycore_bitutils.h index 0bd3270fe82e5..1602fc68d9407 100644 --- a/Include/internal/pycore_bitutils.h +++ b/Include/internal/pycore_bitutils.h @@ -17,10 +17,12 @@ extern "C" { # error "this header requires Py_BUILD_CORE define" #endif -#if defined(__clang__) || \ - (defined(__GNUC__) && \ - ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 8))) - /* __builtin_bswap16() is available since GCC 4.8, +#if ((defined(__GNUC__) \ + && ((__GNUC__ >= 5) || (__GNUC__ == 4) && (__GNUC_MINOR__ >= 8))) \ + || (defined(__clang__) \ + && (__clang_major__ >= 4 \ + || (__clang_major__ == 3 && __clang_minor__ >= 2)))) + /* __builtin_bswap16() is available since GCC 4.8 and clang 3.2, __builtin_bswap32() is available since GCC 4.3, __builtin_bswap64() is available since GCC 4.3. */ # define _PY_HAVE_BUILTIN_BSWAP diff --git a/Misc/NEWS.d/next/Build/2020-08-24-18-34-01.bpo-41617.sKKXz7.rst b/Misc/NEWS.d/next/Build/2020-08-24-18-34-01.bpo-41617.sKKXz7.rst new file mode 100644 index 0000000000000..715eadbee896f --- /dev/null +++ b/Misc/NEWS.d/next/Build/2020-08-24-18-34-01.bpo-41617.sKKXz7.rst @@ -0,0 +1,2 @@ +Fix ``pycore_bitutils.h`` header file to support old clang versions: +``__builtin_bswap16()`` is not available in LLVM clang 3.0.

1 0

bpo-41681: Fix for `f-string/str.format` error description when using 2 `, ` in format specifier (GH-22036)
by han-solo 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/0d6aa7f0ee38eb453bc8f73bf4830e6172… commit: 0d6aa7f0ee38eb453bc8f73bf4830e6172be2f35 branch: master author: han-solo <hanish0019(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-01T10:34:29-04:00 summary: bpo-41681: Fix for `f-string/str.format` error description when using 2 `,` in format specifier (GH-22036) * Fixed `f-string/str.format` error description when using two `,` in format specifier. Co-authored-by: millefalcon <hanish0019(a)hmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2020-08-31-17-49-02.bpo-41681.3-VJiH.rst M Lib/test/test_format.py M Lib/test/test_fstring.py M Python/formatter_unicode.c diff --git a/Lib/test/test_format.py b/Lib/test/test_format.py index e9e5bb9cf00a6..dff8c69032187 100644 --- a/Lib/test/test_format.py +++ b/Lib/test/test_format.py @@ -1,6 +1,7 @@ from test.support import verbose, TestFailed import locale import sys +import re import test.support as support import unittest @@ -495,6 +496,25 @@ def test_g_format_has_no_trailing_zeros(self): self.assertEqual(format(12300050.0, ".6g"), "1.23e+07") self.assertEqual(format(12300050.0, "#.6g"), "1.23000e+07") + def test_with_two_commas_in_format_specifier(self): + error_msg = re.escape("Cannot specify ',' with ','.") + with self.assertRaisesRegex(ValueError, error_msg): + '{:,,}'.format(1) + + def test_with_two_underscore_in_format_specifier(self): + error_msg = re.escape("Cannot specify '_' with '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + '{:__}'.format(1) + + def test_with_a_commas_and_an_underscore_in_format_specifier(self): + error_msg = re.escape("Cannot specify both ',' and '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + '{:,_}'.format(1) + + def test_with_an_underscore_and_a_comma_in_format_specifier(self): + error_msg = re.escape("Cannot specify both ',' and '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + '{:,_}'.format(1) if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_fstring.py b/Lib/test/test_fstring.py index 35a62a0632e2e..b9bede0d9b800 100644 --- a/Lib/test/test_fstring.py +++ b/Lib/test/test_fstring.py @@ -9,6 +9,7 @@ import ast import os +import re import types import decimal import unittest @@ -1198,6 +1199,25 @@ def test_invalid_syntax_error_message(self): with self.assertRaisesRegex(SyntaxError, "f-string: invalid syntax"): compile("f'{a $ b}'", "?", "exec") + def test_with_two_commas_in_format_specifier(self): + error_msg = re.escape("Cannot specify ',' with ','.") + with self.assertRaisesRegex(ValueError, error_msg): + f'{1:,,}' + + def test_with_two_underscore_in_format_specifier(self): + error_msg = re.escape("Cannot specify '_' with '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + f'{1:__}' + + def test_with_a_commas_and_an_underscore_in_format_specifier(self): + error_msg = re.escape("Cannot specify both ',' and '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + f'{1:,_}' + + def test_with_an_underscore_and_a_comma_in_format_specifier(self): + error_msg = re.escape("Cannot specify both ',' and '_'.") + with self.assertRaisesRegex(ValueError, error_msg): + f'{1:,_}' if __name__ == '__main__': unittest.main() diff --git a/Misc/NEWS.d/next/Core and Builtins/2020-08-31-17-49-02.bpo-41681.3-VJiH.rst b/Misc/NEWS.d/next/Core and Builtins/2020-08-31-17-49-02.bpo-41681.3-VJiH.rst new file mode 100644 index 0000000000000..ed557f92d85ca --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2020-08-31-17-49-02.bpo-41681.3-VJiH.rst @@ -0,0 +1,2 @@ +Fixes the wrong error description in the error raised by using 2 `,` in +format string in f-string and :meth:`str.format`. diff --git a/Python/formatter_unicode.c b/Python/formatter_unicode.c index 74638ca223772..ed95f267d476c 100644 --- a/Python/formatter_unicode.c +++ b/Python/formatter_unicode.c @@ -252,8 +252,10 @@ parse_internal_render_format_spec(PyObject *format_spec, ++pos; } if (end-pos && READ_spec(pos) == ',') { - invalid_comma_and_underscore(); - return 0; + if (format->thousands_separators == LT_UNDERSCORE_LOCALE) { + invalid_comma_and_underscore(); + return 0; + } } /* Parse field precision */

1 0

[doc] Add link to FileHandler in logging (GH-21940)
by Andre Delfino 01 Sep '20

01 Sep '20

1 0

bpo-41685: Temporarily pin setuptools to 49.2.1 in Docs venv. (GH-22038)
by Ned Deily 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/a4c4e17f959621ab00a98086af1704fb0c… commit: a4c4e17f959621ab00a98086af1704fb0c896466 branch: master author: Ned Deily <nad(a)python.org> committer: GitHub <noreply(a)github.com> date: 2020-09-01T05:40:27-04:00 summary: bpo-41685: Temporarily pin setuptools to 49.2.1 in Docs venv. (GH-22038) See https://github.com/pypa/setuptools/pull/2361 files: M Doc/Makefile diff --git a/Doc/Makefile b/Doc/Makefile index b8ca1edfbc60a..87d5d07665c86 100644 --- a/Doc/Makefile +++ b/Doc/Makefile @@ -142,7 +142,8 @@ clean: venv: $(PYTHON) -m venv $(VENVDIR) - $(VENVDIR)/bin/python3 -m pip install -U pip setuptools +# $(VENVDIR)/bin/python3 -m pip install -U pip setuptools + $(VENVDIR)/bin/python3 -m pip install -U pip setuptools==49.2.1 $(VENVDIR)/bin/python3 -m pip install -U Sphinx==2.3.1 blurb python-docs-theme @echo "The venv has been created in the $(VENVDIR) directory"

1 0

[doc] Document VIRTUAL_ENV environment variable (GH-21970)
by Andre Delfino 01 Sep '20

01 Sep '20

https://github.com/python/cpython/commit/3584d4b64a5373440f78237eac734831cf… commit: 3584d4b64a5373440f78237eac734831cfd83f79 branch: master author: Andre Delfino <adelfino(a)gmail.com> committer: GitHub <noreply(a)github.com> date: 2020-09-01T07:07:29+01:00 summary: [doc] Document VIRTUAL_ENV environment variable (GH-21970) files: M Doc/using/venv-create.inc diff --git a/Doc/using/venv-create.inc b/Doc/using/venv-create.inc index c8f6e8f87d567..8f850a74d413c 100644 --- a/Doc/using/venv-create.inc +++ b/Doc/using/venv-create.inc @@ -126,6 +126,10 @@ directory containing the virtual environment): | | PowerShell | PS C:\\> <venv>\\Scripts\\Activate.ps1 | +-------------+-----------------+-----------------------------------------+ +When a virtual environment is active, the :envvar:`VIRTUAL_ENV` environment +variable is set to the path of the virtual environment. This can be used to +check if one is running inside a virtual environment. + You don't specifically *need* to activate an environment; activation just prepends the virtual environment's binary directory to your path, so that "python" invokes the virtual environment's Python interpreter and you can run

1 0