Mailman 3 June 2021 - Python-checkins

bpo-44320: Fix markup for W3C C14N test suite (GH-26556)
by corona10 June 5, 2021

June 5, 2021

https://github.com/python/cpython/commit/71be46170490d08743c714b9fa4484038a… commit: 71be46170490d08743c714b9fa4484038aa7a23e branch: main author: NAKAMURA Osamu <osamu0329nakamura(a)users.noreply.github.com> committer: corona10 <donghee.na92(a)gmail.com> date: 2021-06-06T11:34:10+09:00 summary: bpo-44320: Fix markup for W3C C14N test suite (GH-26556) files: M Doc/license.rst diff --git a/Doc/license.rst b/Doc/license.rst index f487d98b2b43a..d459ff664ceed 100644 --- a/Doc/license.rst +++ b/Doc/license.rst @@ -955,7 +955,7 @@ W3C C14N test suite The C14N 2.0 test suite in the :mod:`test` package (``Lib/test/xmltestdata/c14n-20/``) was retrieved from the W3C website at https://www.w3.org/TR/xml-c14n2-testcases/ and is distributed under the -3-clause BSD license: +3-clause BSD license:: Copyright (c) 2013 W3C(R) (MIT, ERCIM, Keio, Beihang), All Rights Reserved.

1 0

bpo-44304: Ensure the sqlite3 destructor callback is always called with the GIL held (GH-26551) (GH_26552)
by pablogsal June 5, 2021

June 5, 2021

https://github.com/python/cpython/commit/317e9ed4363a86b1364573c5a5e30011a0… commit: 317e9ed4363a86b1364573c5a5e30011a080ce6d branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-06-06T00:13:27+01:00 summary: bpo-44304: Ensure the sqlite3 destructor callback is always called with the GIL held (GH-26551) (GH_26552) (cherry picked from commit 6e3b7cf3af3ed7758b2c2193c1d393feb8ab8f72) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: M Modules/_sqlite/connection.c M Modules/_sqlite/statement.c diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index fccffabc4b2a0..8e42a36bd3372 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -851,7 +851,12 @@ static void _pysqlite_drop_unused_cursor_references(pysqlite_Connection* self) static void _destructor(void* args) { + // This function may be called without the GIL held, so we need to ensure + // that we destroy 'args' with the GIL + PyGILState_STATE gstate; + gstate = PyGILState_Ensure(); Py_DECREF((PyObject*)args); + PyGILState_Release(gstate); } /*[clinic input] diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c index cf7fba6ce9a24..072b07d4eaba5 100644 --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -398,7 +398,9 @@ stmt_dealloc(pysqlite_Statement *self) PyObject_ClearWeakRefs((PyObject*)self); } if (self->st) { + Py_BEGIN_ALLOW_THREADS sqlite3_finalize(self->st); + Py_END_ALLOW_THREADS self->st = 0; } tp->tp_clear((PyObject *)self);

1 0

bpo-44304: Ensure the sqlite3 destructor callback is always called with the GIL held (GH-26551)
by pablogsal June 5, 2021

June 5, 2021

https://github.com/python/cpython/commit/6e3b7cf3af3ed7758b2c2193c1d393feb8… commit: 6e3b7cf3af3ed7758b2c2193c1d393feb8ab8f72 branch: main author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-06-05T23:41:11+01:00 summary: bpo-44304: Ensure the sqlite3 destructor callback is always called with the GIL held (GH-26551) files: M Modules/_sqlite/connection.c M Modules/_sqlite/statement.c diff --git a/Modules/_sqlite/connection.c b/Modules/_sqlite/connection.c index c2f3bc0f20125..fbee1b7ce52a2 100644 --- a/Modules/_sqlite/connection.c +++ b/Modules/_sqlite/connection.c @@ -825,7 +825,12 @@ static void _pysqlite_drop_unused_cursor_references(pysqlite_Connection* self) static void _destructor(void* args) { + // This function may be called without the GIL held, so we need to ensure + // that we destroy 'args' with the GIL + PyGILState_STATE gstate; + gstate = PyGILState_Ensure(); Py_DECREF((PyObject*)args); + PyGILState_Release(gstate); } /*[clinic input] diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c index c9dd882945328..f12ef67d261c7 100644 --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -404,7 +404,9 @@ stmt_dealloc(pysqlite_Statement *self) PyObject_ClearWeakRefs((PyObject*)self); } if (self->st) { + Py_BEGIN_ALLOW_THREADS sqlite3_finalize(self->st); + Py_END_ALLOW_THREADS self->st = 0; } tp->tp_clear((PyObject *)self);

1 0

bpo-44304: Fix crash in the sqlite3 module when the GC clears Statement objects (GH-26545)
by miss-islington June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/ad2f3b74b5615aa36a82d1fdbc45bb7468… commit: ad2f3b74b5615aa36a82d1fdbc45bb7468aa1d72 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2021-06-04T20:09:40-07:00 summary: bpo-44304: Fix crash in the sqlite3 module when the GC clears Statement objects (GH-26545) (cherry picked from commit fa106a685c1f199aca5be5c2d0277a14cc9057bd) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: A Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst M Lib/sqlite3/test/userfunctions.py M Modules/_sqlite/statement.c diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py index 6f57d1911b24d8..429089072496ed 100644 --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -23,6 +23,7 @@ import unittest import unittest.mock +import gc import sqlite3 as sqlite def func_returntext(): @@ -322,6 +323,22 @@ def test_func_deterministic_keyword_only(self): with self.assertRaises(TypeError): self.con.create_function("deterministic", 0, int, True) + def test_function_destructor_via_gc(self): + # See bpo-44304: The destructor of the user function can + # crash if is called without the GIL from the gc functions + dest = sqlite.connect(':memory:') + def md5sum(t): + return + + dest.create_function("md5", 1, md5sum) + x = dest("create table lang (name, first_appeared)") + del md5sum, dest + + y = [x] + y.append(y) + + del x,y + gc.collect() class AggregateTests(unittest.TestCase): def setUp(self): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst new file mode 100644 index 00000000000000..89104e8e387ed1 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst @@ -0,0 +1,2 @@ +Fix a crash in the :mod:`sqlite3` module that happened when the garbage +collector clears :class:`sqlite.Statement` objects. Patch by Pablo Galindo diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c index e6dc4fd89528d2..cf7fba6ce9a24c 100644 --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -397,6 +397,10 @@ stmt_dealloc(pysqlite_Statement *self) if (self->in_weakreflist != NULL) { PyObject_ClearWeakRefs((PyObject*)self); } + if (self->st) { + sqlite3_finalize(self->st); + self->st = 0; + } tp->tp_clear((PyObject *)self); tp->tp_free(self); Py_DECREF(tp); @@ -405,13 +409,6 @@ stmt_dealloc(pysqlite_Statement *self) static int stmt_clear(pysqlite_Statement *self) { - if (self->st) { - Py_BEGIN_ALLOW_THREADS - sqlite3_finalize(self->st); - Py_END_ALLOW_THREADS - self->st = 0; - } - Py_CLEAR(self->sql); return 0; }

1 0

bpo-44304: Fix crash in the sqlite3 module when the GC clears Statement objects (GH-26545)
by pablogsal June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/fa106a685c1f199aca5be5c2d0277a14cc… commit: fa106a685c1f199aca5be5c2d0277a14cc9057bd branch: main author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-06-05T03:50:39+01:00 summary: bpo-44304: Fix crash in the sqlite3 module when the GC clears Statement objects (GH-26545) files: A Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst M Lib/sqlite3/test/userfunctions.py M Modules/_sqlite/statement.c diff --git a/Lib/sqlite3/test/userfunctions.py b/Lib/sqlite3/test/userfunctions.py index 6f57d1911b24d..429089072496e 100644 --- a/Lib/sqlite3/test/userfunctions.py +++ b/Lib/sqlite3/test/userfunctions.py @@ -23,6 +23,7 @@ import unittest import unittest.mock +import gc import sqlite3 as sqlite def func_returntext(): @@ -322,6 +323,22 @@ def test_func_deterministic_keyword_only(self): with self.assertRaises(TypeError): self.con.create_function("deterministic", 0, int, True) + def test_function_destructor_via_gc(self): + # See bpo-44304: The destructor of the user function can + # crash if is called without the GIL from the gc functions + dest = sqlite.connect(':memory:') + def md5sum(t): + return + + dest.create_function("md5", 1, md5sum) + x = dest("create table lang (name, first_appeared)") + del md5sum, dest + + y = [x] + y.append(y) + + del x,y + gc.collect() class AggregateTests(unittest.TestCase): def setUp(self): diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst b/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst new file mode 100644 index 0000000000000..89104e8e387ed --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-06-05-02-34-57.bpo-44304._MAoPc.rst @@ -0,0 +1,2 @@ +Fix a crash in the :mod:`sqlite3` module that happened when the garbage +collector clears :class:`sqlite.Statement` objects. Patch by Pablo Galindo diff --git a/Modules/_sqlite/statement.c b/Modules/_sqlite/statement.c index c86645ad42b64..c9dd882945328 100644 --- a/Modules/_sqlite/statement.c +++ b/Modules/_sqlite/statement.c @@ -403,6 +403,10 @@ stmt_dealloc(pysqlite_Statement *self) if (self->in_weakreflist != NULL) { PyObject_ClearWeakRefs((PyObject*)self); } + if (self->st) { + sqlite3_finalize(self->st); + self->st = 0; + } tp->tp_clear((PyObject *)self); tp->tp_free(self); Py_DECREF(tp); @@ -411,13 +415,6 @@ stmt_dealloc(pysqlite_Statement *self) static int stmt_clear(pysqlite_Statement *self) { - if (self->st) { - Py_BEGIN_ALLOW_THREADS - sqlite3_finalize(self->st); - Py_END_ALLOW_THREADS - self->st = 0; - } - Py_CLEAR(self->sql); return 0; }

1 0

Update nonstandard variable names (GH-26540) (GH-26546)
by rhettinger June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/4642caf232a3f01468e76f19cd0c88175e… commit: 4642caf232a3f01468e76f19cd0c88175e10ee28 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2021-06-04T19:38:30-07:00 summary: Update nonstandard variable names (GH-26540) (GH-26546) files: M Lib/statistics.py diff --git a/Lib/statistics.py b/Lib/statistics.py index ceb8af81b0d3d..268cc71a0952b 100644 --- a/Lib/statistics.py +++ b/Lib/statistics.py @@ -913,10 +913,10 @@ def correlation(x, y, /): xbar = fsum(x) / n ybar = fsum(y) / n sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) - s2x = fsum((xi - xbar) ** 2.0 for xi in x) - s2y = fsum((yi - ybar) ** 2.0 for yi in y) + sxx = fsum((xi - xbar) ** 2.0 for xi in x) + syy = fsum((yi - ybar) ** 2.0 for yi in y) try: - return sxy / sqrt(s2x * s2y) + return sxy / sqrt(sxx * syy) except ZeroDivisionError: raise StatisticsError('at least one of the inputs is constant') @@ -957,9 +957,9 @@ def linear_regression(x, y, /): xbar = fsum(x) / n ybar = fsum(y) / n sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) - s2x = fsum((xi - xbar) ** 2.0 for xi in x) + sxx = fsum((xi - xbar) ** 2.0 for xi in x) try: - slope = sxy / s2x # equivalent to: covariance(x, y) / variance(x) + slope = sxy / sxx # equivalent to: covariance(x, y) / variance(x) except ZeroDivisionError: raise StatisticsError('x is constant') intercept = ybar - slope * xbar

1 0

Improve speed and accuracy for correlation() (GH-26135) (GH-26151)
by rhettinger June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/5442cfa67b6ddf41daaf93b322942da3d2… commit: 5442cfa67b6ddf41daaf93b322942da3d20d2402 branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: rhettinger <rhettinger(a)users.noreply.github.com> date: 2021-06-04T18:49:29-07:00 summary: Improve speed and accuracy for correlation() (GH-26135) (GH-26151) files: M Lib/statistics.py diff --git a/Lib/statistics.py b/Lib/statistics.py index b1b613171d2c8..ceb8af81b0d3d 100644 --- a/Lib/statistics.py +++ b/Lib/statistics.py @@ -107,9 +107,12 @@ __all__ = [ 'NormalDist', 'StatisticsError', + 'correlation', + 'covariance', 'fmean', 'geometric_mean', 'harmonic_mean', + 'linear_regression', 'mean', 'median', 'median_grouped', @@ -122,9 +125,6 @@ 'quantiles', 'stdev', 'variance', - 'correlation', - 'covariance', - 'linear_regression', ] import math @@ -882,10 +882,10 @@ def covariance(x, y, /): raise StatisticsError('covariance requires that both inputs have same number of data points') if n < 2: raise StatisticsError('covariance requires at least two data points') - xbar = fmean(x) - ybar = fmean(y) - total = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) - return total / (n - 1) + xbar = fsum(x) / n + ybar = fsum(y) / n + sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) + return sxy / (n - 1) def correlation(x, y, /): @@ -910,11 +910,13 @@ def correlation(x, y, /): raise StatisticsError('correlation requires that both inputs have same number of data points') if n < 2: raise StatisticsError('correlation requires at least two data points') - cov = covariance(x, y) - stdx = stdev(x) - stdy = stdev(y) + xbar = fsum(x) / n + ybar = fsum(y) / n + sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) + s2x = fsum((xi - xbar) ** 2.0 for xi in x) + s2y = fsum((yi - ybar) ** 2.0 for yi in y) try: - return cov / (stdx * stdy) + return sxy / sqrt(s2x * s2y) except ZeroDivisionError: raise StatisticsError('at least one of the inputs is constant') @@ -957,7 +959,7 @@ def linear_regression(x, y, /): sxy = fsum((xi - xbar) * (yi - ybar) for xi, yi in zip(x, y)) s2x = fsum((xi - xbar) ** 2.0 for xi in x) try: - slope = sxy / s2x + slope = sxy / s2x # equivalent to: covariance(x, y) / variance(x) except ZeroDivisionError: raise StatisticsError('x is constant') intercept = ybar - slope * xbar

1 0

bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542) (GH-26544)
by pablogsal June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/0d441d2e70e365b5dc517d8ff24a20b97b… commit: 0d441d2e70e365b5dc517d8ff24a20b97bc4536a branch: 3.10 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-06-05T01:05:54+01:00 summary: bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542) (GH-26544) (cherry picked from commit f171877ebe276749f31386baed5841ce37cbee2e) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: M Lib/test/test_asyncio/test_subprocess.py diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 225a3babc844b8..3cf88188ecf3b1 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -665,6 +665,8 @@ class SubprocessThreadedWatcherTests(SubprocessWatcherMixin, Watcher = unix_events.ThreadedChildWatcher + @unittest.skip("bpo-38323: MultiLoopChildWatcher has a race condition \ + and these tests can hang the test suite") class SubprocessMultiLoopWatcherTests(SubprocessWatcherMixin, test_utils.TestCase):

1 0

bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542)
by miss-islington June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/b3c50b29e14d8f089aee3e0980298bfc74… commit: b3c50b29e14d8f089aee3e0980298bfc74cb3ba7 branch: 3.9 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: miss-islington <31488909+miss-islington(a)users.noreply.github.com> date: 2021-06-04T16:59:55-07:00 summary: bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542) (cherry picked from commit f171877ebe276749f31386baed5841ce37cbee2e) Co-authored-by: Pablo Galindo <Pablogsal(a)gmail.com> files: M Lib/test/test_asyncio/test_subprocess.py diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 6657a88e657c25..693cc412904307 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -684,6 +684,8 @@ class SubprocessThreadedWatcherTests(SubprocessWatcherMixin, Watcher = unix_events.ThreadedChildWatcher + @unittest.skip("bpo-38323: MultiLoopChildWatcher has a race condition \ + and these tests can hang the test suite") class SubprocessMultiLoopWatcherTests(SubprocessWatcherMixin, test_utils.TestCase):

1 0

bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542)
by pablogsal June 4, 2021

June 4, 2021

https://github.com/python/cpython/commit/f171877ebe276749f31386baed5841ce37… commit: f171877ebe276749f31386baed5841ce37cbee2e branch: main author: Pablo Galindo <Pablogsal(a)gmail.com> committer: pablogsal <Pablogsal(a)gmail.com> date: 2021-06-05T00:33:20+01:00 summary: bpo-38323: Skip SubprocessMultiLoopWatcherTest as they can hang the test suite (GH-26542) files: M Lib/test/test_asyncio/test_subprocess.py diff --git a/Lib/test/test_asyncio/test_subprocess.py b/Lib/test/test_asyncio/test_subprocess.py index 225a3babc844b8..3cf88188ecf3b1 100644 --- a/Lib/test/test_asyncio/test_subprocess.py +++ b/Lib/test/test_asyncio/test_subprocess.py @@ -665,6 +665,8 @@ class SubprocessThreadedWatcherTests(SubprocessWatcherMixin, Watcher = unix_events.ThreadedChildWatcher + @unittest.skip("bpo-38323: MultiLoopChildWatcher has a race condition \ + and these tests can hang the test suite") class SubprocessMultiLoopWatcherTests(SubprocessWatcherMixin, test_utils.TestCase):

1 0