Mailman 3 April 2019 - Python-checkins

bpo-36763: Fix Py_SetStandardStreamEncoding() (GH-13028)
by Victor Stinner 01 May '19

01 May '19

https://github.com/python/cpython/commit/463b82a3efe8a6a9f3924a5b37482e961d… commit: 463b82a3efe8a6a9f3924a5b37482e961dffe3b8 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-05-01T01:36:13+02:00 summary: bpo-36763: Fix Py_SetStandardStreamEncoding() (GH-13028) Fix memory leak in Py_SetStandardStreamEncoding(): release memory if the function is called twice. files: A Misc/NEWS.d/next/C API/2019-05-01-00-42-08.bpo-36763.vghb86.rst M Python/coreconfig.c diff --git a/Misc/NEWS.d/next/C API/2019-05-01-00-42-08.bpo-36763.vghb86.rst b/Misc/NEWS.d/next/C API/2019-05-01-00-42-08.bpo-36763.vghb86.rst new file mode 100644 index 000000000000..1c34920827fc --- /dev/null +++ b/Misc/NEWS.d/next/C API/2019-05-01-00-42-08.bpo-36763.vghb86.rst @@ -0,0 +1,2 @@ +Fix memory leak in :c:func:`Py_SetStandardStreamEncoding`: release memory if +the function is called twice. diff --git a/Python/coreconfig.c b/Python/coreconfig.c index d05beef2aa55..471d5126f803 100644 --- a/Python/coreconfig.c +++ b/Python/coreconfig.c @@ -375,6 +375,7 @@ Py_SetStandardStreamEncoding(const char *encoding, const char *errors) * Py_Initialize hasn't been called yet. */ if (encoding) { + PyMem_RawFree(_Py_StandardStreamEncoding); _Py_StandardStreamEncoding = _PyMem_RawStrdup(encoding); if (!_Py_StandardStreamEncoding) { res = -2; @@ -382,11 +383,11 @@ Py_SetStandardStreamEncoding(const char *encoding, const char *errors) } } if (errors) { + PyMem_RawFree(_Py_StandardStreamErrors); _Py_StandardStreamErrors = _PyMem_RawStrdup(errors); if (!_Py_StandardStreamErrors) { - if (_Py_StandardStreamEncoding) { - PyMem_RawFree(_Py_StandardStreamEncoding); - } + PyMem_RawFree(_Py_StandardStreamEncoding); + _Py_StandardStreamEncoding = NULL; res = -3; goto done; }

1 0

bpo-36734: Fix compilation of faulthandler.c on HP-UX (GH-12970)
by Miss Islington (bot) 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/5f5b187bfa17254f5ae55593820fc938c4… commit: 5f5b187bfa17254f5ae55593820fc938c45c2b32 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-04-30T14:30:44-07:00 summary: bpo-36734: Fix compilation of faulthandler.c on HP-UX (GH-12970) Initialize "stack_t current_stack" to zero using memset(). (cherry picked from commit b84cb70880a0acfcbbaca7bcda405af08f94d269) Co-authored-by: Victor Stinner <vstinner(a)redhat.com> files: A Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst M Modules/faulthandler.c diff --git a/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst b/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst new file mode 100644 index 000000000000..09341990a63d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst @@ -0,0 +1,2 @@ +Fix compilation of ``faulthandler.c`` on HP-UX. Initialize ``stack_t +current_stack`` to zero using ``memset()``. diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index cf24c9b2b9ad..ec5192832ce0 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -1369,7 +1369,8 @@ void _PyFaulthandler_Fini(void) #ifdef HAVE_SIGALTSTACK if (stack.ss_sp != NULL) { /* Fetch the current alt stack */ - stack_t current_stack = {}; + stack_t current_stack; + memset(&current_stack, 0, sizeof(current_stack)); if (sigaltstack(NULL, &current_stack) == 0) { if (current_stack.ss_sp == stack.ss_sp) { /* The current alt stack is the one that we installed.

1 0

Don't report deleted attributes in __dir__ (GH#10148)
by Chris Withers 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/0df635c7f8aa69e56a092bd4f142f0f164… commit: 0df635c7f8aa69e56a092bd4f142f0f164741ab2 branch: master author: Mario Corchero <mariocj89(a)gmail.com> committer: Chris Withers <chris(a)withers.org> date: 2019-04-30T19:56:36+01:00 summary: Don't report deleted attributes in __dir__ (GH#10148) When an attribute is deleted from a Mock, a sentinel is added rather than just deleting the attribute. This commit checks for such sentinels when returning the child mocks in the __dir__ method as users won't expect deleted attributes to appear when performing dir(mock). files: A Misc/NEWS.d/next/Library/2018-10-27-11-54-12.bpo-35082.HDj1nr.rst M Lib/unittest/mock.py M Lib/unittest/test/testmock/testmock.py diff --git a/Lib/unittest/mock.py b/Lib/unittest/mock.py index 1636073ff009..997af7172566 100644 --- a/Lib/unittest/mock.py +++ b/Lib/unittest/mock.py @@ -684,12 +684,14 @@ def __dir__(self): extras = self._mock_methods or [] from_type = dir(type(self)) from_dict = list(self.__dict__) + from_child_mocks = [ + m_name for m_name, m_value in self._mock_children.items() + if m_value is not _deleted] from_type = [e for e in from_type if not e.startswith('_')] from_dict = [e for e in from_dict if not e.startswith('_') or _is_magic(e)] - return sorted(set(extras + from_type + from_dict + - list(self._mock_children))) + return sorted(set(extras + from_type + from_dict + from_child_mocks)) def __setattr__(self, name, value): diff --git a/Lib/unittest/test/testmock/testmock.py b/Lib/unittest/test/testmock/testmock.py index bdaebbe66b74..0e7e4a1d8c93 100644 --- a/Lib/unittest/test/testmock/testmock.py +++ b/Lib/unittest/test/testmock/testmock.py @@ -885,6 +885,15 @@ def test_filter_dir(self): patcher.stop() + def test_dir_does_not_include_deleted_attributes(self): + mock = Mock() + mock.child.return_value = 1 + + self.assertIn('child', dir(mock)) + del mock.child + self.assertNotIn('child', dir(mock)) + + def test_configure_mock(self): mock = Mock(foo='bar') self.assertEqual(mock.foo, 'bar') diff --git a/Misc/NEWS.d/next/Library/2018-10-27-11-54-12.bpo-35082.HDj1nr.rst b/Misc/NEWS.d/next/Library/2018-10-27-11-54-12.bpo-35082.HDj1nr.rst new file mode 100644 index 000000000000..45a0729506e3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-10-27-11-54-12.bpo-35082.HDj1nr.rst @@ -0,0 +1,2 @@ +Don't return deleted attributes when calling dir on a +:class:`unittest.mock.Mock`.

1 0

bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
by Miss Islington (bot) 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/4d723e76e1ad17e9e7d5e828e59bb47e76… commit: 4d723e76e1ad17e9e7d5e828e59bb47e76f2174b branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-04-30T05:21:02-07:00 summary: bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) (cherry picked from commit d537ab0ff9767ef024f26246899728f0116b1ec3) Co-authored-by: Steve Dower <steve.dower(a)python.org> files: A Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst M Lib/test/test_urlparse.py M Lib/urllib/parse.py diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py index e6638aee2244..c26235449461 100644 --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -1001,6 +1001,12 @@ def test_urlsplit_normalization(self): self.assertIn('\u2100', denorm_chars) self.assertIn('\uFF03', denorm_chars) + # bpo-36742: Verify port separators are ignored when they + # existed prior to decomposition + urllib.parse.urlsplit('http://\u30d5\u309a:80') + with self.assertRaises(ValueError): + urllib.parse.urlsplit('http://\u30d5\u309a\ufe1380') + for scheme in ["http", "https", "ftp"]: for c in denorm_chars: url = "{}://netloc{}false.netloc/path".format(scheme, c) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index 1eec26e0f1f3..f5b3487ea9d6 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -397,13 +397,16 @@ def _checknetloc(netloc): # looking for characters like \u2100 that expand to 'a/c' # IDNA uses NFKC equivalence, so normalize for this check import unicodedata - netloc2 = unicodedata.normalize('NFKC', netloc) - if netloc == netloc2: + n = netloc.rpartition('@')[2] # ignore anything to the left of '@' + n = n.replace(':', '') # ignore characters already included + n = n.replace('#', '') # but not the surrounding text + n = n.replace('?', '') + netloc2 = unicodedata.normalize('NFKC', n) + if n == netloc2: return - _, _, netloc = netloc.rpartition('@') # anything to the left of '@' is okay for c in '/?#@:': if c in netloc2: - raise ValueError("netloc '" + netloc2 + "' contains invalid " + + raise ValueError("netloc '" + netloc + "' contains invalid " + "characters under NFKC normalization") def urlsplit(url, scheme='', allow_fragments=True): diff --git a/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst b/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst new file mode 100644 index 000000000000..d729ed2f3cd5 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst @@ -0,0 +1 @@ +Fixes mishandling of pre-normalization characters in urlsplit().

1 0

bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017)
by Steve Dower 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/d537ab0ff9767ef024f26246899728f011… commit: d537ab0ff9767ef024f26246899728f0116b1ec3 branch: master author: Steve Dower <steve.dower(a)python.org> committer: GitHub <noreply(a)github.com> date: 2019-04-30T12:03:02Z summary: bpo-36742: Fixes handling of pre-normalization characters in urlsplit() (GH-13017) files: A Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst M Lib/test/test_urlparse.py M Lib/urllib/parse.py diff --git a/Lib/test/test_urlparse.py b/Lib/test/test_urlparse.py index 0faf2bbb6459..d0365ecab72c 100644 --- a/Lib/test/test_urlparse.py +++ b/Lib/test/test_urlparse.py @@ -1011,6 +1011,12 @@ def test_urlsplit_normalization(self): self.assertIn('\u2100', denorm_chars) self.assertIn('\uFF03', denorm_chars) + # bpo-36742: Verify port separators are ignored when they + # existed prior to decomposition + urllib.parse.urlsplit('http://\u30d5\u309a:80') + with self.assertRaises(ValueError): + urllib.parse.urlsplit('http://\u30d5\u309a\ufe1380') + for scheme in ["http", "https", "ftp"]: for c in denorm_chars: url = "{}://netloc{}false.netloc/path".format(scheme, c) diff --git a/Lib/urllib/parse.py b/Lib/urllib/parse.py index fb518a97749c..dfba704144e9 100644 --- a/Lib/urllib/parse.py +++ b/Lib/urllib/parse.py @@ -402,13 +402,16 @@ def _checknetloc(netloc): # looking for characters like \u2100 that expand to 'a/c' # IDNA uses NFKC equivalence, so normalize for this check import unicodedata - netloc2 = unicodedata.normalize('NFKC', netloc) - if netloc == netloc2: + n = netloc.rpartition('@')[2] # ignore anything to the left of '@' + n = n.replace(':', '') # ignore characters already included + n = n.replace('#', '') # but not the surrounding text + n = n.replace('?', '') + netloc2 = unicodedata.normalize('NFKC', n) + if n == netloc2: return - _, _, netloc = netloc.rpartition('@') # anything to the left of '@' is okay for c in '/?#@:': if c in netloc2: - raise ValueError("netloc '" + netloc2 + "' contains invalid " + + raise ValueError("netloc '" + netloc + "' contains invalid " + "characters under NFKC normalization") def urlsplit(url, scheme='', allow_fragments=True): diff --git a/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst b/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst new file mode 100644 index 000000000000..d729ed2f3cd5 --- /dev/null +++ b/Misc/NEWS.d/next/Security/2019-04-29-15-34-59.bpo-36742.QCUY0i.rst @@ -0,0 +1 @@ +Fixes mishandling of pre-normalization characters in urlsplit().

1 0

bpo-36734: Fix compilation of faulthandler.c on HP-UX (GH-12970)
by Victor Stinner 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/b84cb70880a0acfcbbaca7bcda405af08f… commit: b84cb70880a0acfcbbaca7bcda405af08f94d269 branch: master author: Victor Stinner <vstinner(a)redhat.com> committer: GitHub <noreply(a)github.com> date: 2019-04-30T12:19:34+02:00 summary: bpo-36734: Fix compilation of faulthandler.c on HP-UX (GH-12970) Initialize "stack_t current_stack" to zero using memset(). files: A Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst M Modules/faulthandler.c diff --git a/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst b/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst new file mode 100644 index 000000000000..09341990a63d --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-04-26-17-14-20.bpo-36734.p2MaiN.rst @@ -0,0 +1,2 @@ +Fix compilation of ``faulthandler.c`` on HP-UX. Initialize ``stack_t +current_stack`` to zero using ``memset()``. diff --git a/Modules/faulthandler.c b/Modules/faulthandler.c index 30fe18695fec..d45b8660ee65 100644 --- a/Modules/faulthandler.c +++ b/Modules/faulthandler.c @@ -1370,7 +1370,8 @@ void _PyFaulthandler_Fini(void) #ifdef HAVE_SIGALTSTACK if (stack.ss_sp != NULL) { /* Fetch the current alt stack */ - stack_t current_stack = {}; + stack_t current_stack; + memset(&current_stack, 0, sizeof(current_stack)); if (sigaltstack(NULL, &current_stack) == 0) { if (current_stack.ss_sp == stack.ss_sp) { /* The current alt stack is the one that we installed.

1 0

bpo-25430: improve performance of IPNetwork.__contains__ (GH-1785)
by Inada Naoki 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/3bbcc92577f8e616bc94c679040043bacd… commit: 3bbcc92577f8e616bc94c679040043bacd00ebf1 branch: master author: gescheit <gescheit12(a)gmail.com> committer: Inada Naoki <songofacandy(a)gmail.com> date: 2019-04-30T16:54:30+09:00 summary: bpo-25430: improve performance of IPNetwork.__contains__ (GH-1785) make a compare in bit-operation manner. files: A Misc/NEWS.d/next/Library/2019-04-15-12-22-09.bpo-25430.7_8kqc.rst M Lib/ipaddress.py diff --git a/Lib/ipaddress.py b/Lib/ipaddress.py index 909a55de4f19..662d73738907 100644 --- a/Lib/ipaddress.py +++ b/Lib/ipaddress.py @@ -697,8 +697,7 @@ def __contains__(self, other): # dealing with another address else: # address - return (int(self.network_address) <= int(other._ip) <= - int(self.broadcast_address)) + return other._ip & self.netmask._ip == self.network_address._ip def overlaps(self, other): """Tell if self is partly contained in other.""" diff --git a/Misc/NEWS.d/next/Library/2019-04-15-12-22-09.bpo-25430.7_8kqc.rst b/Misc/NEWS.d/next/Library/2019-04-15-12-22-09.bpo-25430.7_8kqc.rst new file mode 100644 index 000000000000..922bdef56ec3 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2019-04-15-12-22-09.bpo-25430.7_8kqc.rst @@ -0,0 +1 @@ +improve performance of ``IPNetwork.__contains__()`` \ No newline at end of file

1 0

bpo-36018: Test idempotence. Test two methods against one-another. (GH-13021)
by Raymond Hettinger 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/b0a2c0fa83f9b79616ccf451687096542d… commit: b0a2c0fa83f9b79616ccf451687096542de1e6f8 branch: master author: Raymond Hettinger <rhettinger(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-04-29T23:47:33-07:00 summary: bpo-36018: Test idempotence. Test two methods against one-another. (GH-13021) files: M Lib/test/test_statistics.py diff --git a/Lib/test/test_statistics.py b/Lib/test/test_statistics.py index 0a967055f0fe..903ee8f343cb 100644 --- a/Lib/test/test_statistics.py +++ b/Lib/test/test_statistics.py @@ -2158,6 +2158,20 @@ def test_specific_cases(self): result = quantiles(map(datatype, data), n=n) self.assertTrue(all(type(x) == datatype) for x in result) self.assertEqual(result, list(map(datatype, expected))) + # Quantiles should be idempotent + if len(expected) >= 2: + self.assertEqual(quantiles(expected, n=n), expected) + # Cross-check against other methods + if len(data) >= n: + # After end caps are added, method='inclusive' should + # give the same result as method='exclusive' whenever + # there are more data points than desired cut points. + padded_data = [min(data) - 1000] + data + [max(data) + 1000] + self.assertEqual( + quantiles(data, n=n), + quantiles(padded_data, n=n, method='inclusive'), + (n, data), + ) # Invariant under tranlation and scaling def f(x): return 3.5 * x - 1234.675 @@ -2219,6 +2233,15 @@ def f(x): actual = quantiles(statistics.NormalDist(), n=n, method="inclusive") self.assertTrue(all(math.isclose(e, a, abs_tol=0.0001) for e, a in zip(expected, actual))) + # Whenever n is smaller than the number of data points, running + # method='inclusive' should give the same result as method='exclusive' + # after the two included extreme points are removed. + data = [random.randrange(10_000) for i in range(501)] + actual = quantiles(data, n=32, method='inclusive') + data.remove(min(data)) + data.remove(max(data)) + expected = quantiles(data, n=32) + self.assertEqual(expected, actual) def test_equal_inputs(self): quantiles = statistics.quantiles

1 0

closes bpo-35329: Change 'Package' to 'package' in accordance with PEP8. (GH-13008)
by Miss Islington (bot) 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/4b5340bb634be2ee2a40242cdf4e3f7a0b… commit: 4b5340bb634be2ee2a40242cdf4e3f7a0b6c757a branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-04-29T19:26:00-07:00 summary: closes bpo-35329: Change 'Package' to 'package' in accordance with PEP8. (GH-13008) (cherry picked from commit ee0309f3d83ab9ffa02542bcf45ece84f4fb265e) Co-authored-by: Utkarsh Gupta <guptautkarsh2102(a)gmail.com> files: M Doc/tutorial/modules.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index fd594fd97af4d..d0a68faa2ee25 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -523,7 +523,7 @@ Although certain modules are designed to export only names that follow certain patterns when you use ``import *``, it is still considered bad practice in production code. -Remember, there is nothing wrong with using ``from Package import +Remember, there is nothing wrong with using ``from package import specific_submodule``! In fact, this is the recommended notation unless the importing module needs to use submodules with the same name from different packages.

1 0

closes bpo-35329: Change 'Package' to 'package' in accordance with PEP8. (GH-13008)
by Miss Islington (bot) 30 Apr '19

30 Apr '19

https://github.com/python/cpython/commit/3e5c4a7c804c3ad76a558e5463655c329a… commit: 3e5c4a7c804c3ad76a558e5463655c329aee6437 branch: 2.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2019-04-29T19:25:35-07:00 summary: closes bpo-35329: Change 'Package' to 'package' in accordance with PEP8. (GH-13008) (cherry picked from commit ee0309f3d83ab9ffa02542bcf45ece84f4fb265e) Co-authored-by: Utkarsh Gupta <guptautkarsh2102(a)gmail.com> files: M Doc/tutorial/modules.rst diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst index 042d23306bdd..f767bb69c112 100644 --- a/Doc/tutorial/modules.rst +++ b/Doc/tutorial/modules.rst @@ -518,7 +518,7 @@ Although certain modules are designed to export only names that follow certain patterns when you use ``import *``, it is still considered bad practice in production code. -Remember, there is nothing wrong with using ``from Package import +Remember, there is nothing wrong with using ``from package import specific_submodule``! In fact, this is the recommended notation unless the importing module needs to use submodules with the same name from different packages.

1 0