Mailman 3 March 2018 - Python-checkins

Daily reference leaks (4243df51fe43): sum=4
by solipsis＠pitrou.net March 5, 2018

March 5, 2018

results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [0, 7, -7] memory blocks, sum=0 test_functools leaked [0, 3, 1] memory blocks, sum=4 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogy0oxKo', '--timeout', '7200']

1 0

[3.7] bpo-30147: Add re.escape changes to 3.7 What's New (GH-5978) (GH-5979)
by Ned Deily March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/f92478d57a6b4014dcc0882d43b534fae1… commit: f92478d57a6b4014dcc0882d43b534fae1e7b929 branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: Ned Deily <nad(a)python.org> date: 2018-03-04T18:22:32-05:00 summary: [3.7] bpo-30147: Add re.escape changes to 3.7 What's New (GH-5978) (GH-5979) (cherry picked from commit 18fd89246333bfa1b76c1623df689214f3ce2bf3) Co-authored-by: Cheryl Sabella <cheryl.sabella(a)gmail.com> files: M Doc/whatsnew/3.7.rst diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 46f4f13aab47..9e65488e37d6 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -1214,6 +1214,10 @@ Changes in the Python API (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.) +* Change :func:`re.escape` to only escape regex special characters instead + of escaping all characters other than ASCII letters, numbers, and ``'_'``. + (Contributed by Serhiy Storchaka in :issue:`29995`.) + * :class:`tracemalloc.Traceback` frames are now sorted from oldest to most recent to be more consistent with :mod:`traceback`. (Contributed by Jesse Bakker in :issue:`32121`.)

1 0

bpo-30147: Add re.escape changes to 3.7 What's New (GH-5978)
by Ned Deily March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/18fd89246333bfa1b76c1623df689214f3… commit: 18fd89246333bfa1b76c1623df689214f3ce2bf3 branch: master author: Cheryl Sabella <cheryl.sabella(a)gmail.com> committer: Ned Deily <nad(a)python.org> date: 2018-03-04T18:06:57-05:00 summary: bpo-30147: Add re.escape changes to 3.7 What's New (GH-5978) files: M Doc/whatsnew/3.7.rst diff --git a/Doc/whatsnew/3.7.rst b/Doc/whatsnew/3.7.rst index 46f4f13aab47..9e65488e37d6 100644 --- a/Doc/whatsnew/3.7.rst +++ b/Doc/whatsnew/3.7.rst @@ -1214,6 +1214,10 @@ Changes in the Python API (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.) +* Change :func:`re.escape` to only escape regex special characters instead + of escaping all characters other than ASCII letters, numbers, and ``'_'``. + (Contributed by Serhiy Storchaka in :issue:`29995`.) + * :class:`tracemalloc.Traceback` frames are now sorted from oldest to most recent to be more consistent with :mod:`traceback`. (Contributed by Jesse Bakker in :issue:`32121`.)

1 0

bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701)
by Miss Islington (bot) March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/a5303dd9c263b337f02dda0038f2f5a102… commit: a5303dd9c263b337f02dda0038f2f5a10208140c branch: 3.7 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-03-04T04:00:33-08:00 summary: bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701) (cherry picked from commit 74382a3f175ac285cc924a73fd758e8dc3cc41bb) Co-authored-by: Cheryl Sabella <cheryl.sabella(a)gmail.com> files: A Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst M Lib/tkinter/__init__.py M Lib/tkinter/test/test_tkinter/test_misc.py diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index deea791831ed..53bad3fa95ae 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -739,6 +739,7 @@ def after(self, ms, func=None, *args): if not func: # I'd rather use time.sleep(ms*0.001) self.tk.call('after', ms) + return None else: def callit(): try: @@ -762,11 +763,13 @@ def after_cancel(self, id): """Cancel scheduling of function identified with ID. Identifier returned by after or after_idle must be - given as first parameter.""" + given as first parameter. + """ + if not id: + raise ValueError('id must be a valid identifier returned from ' + 'after or after_idle') try: data = self.tk.call('after', 'info', id) - # In Tk 8.3, splitlist returns: (script, type) - # In Tk 8.4, splitlist may return (script, type) or (script,) script = self.tk.splitlist(data)[0] self.deletecommand(script) except TclError: diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 9dc1e37547fc..1d1a3c29f6bc 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -48,6 +48,114 @@ def test_tk_setPalette(self): '^must specify a background color$', root.tk_setPalette, highlightColor='blue') + def test_after(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Without function, sleeps for ms. + self.assertIsNone(root.after(1)) + + # Set up with callback with no args. + count = 0 + timer1 = root.after(0, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.update() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + timer1 = root.after(0, callback, 42, 11) + root.update() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + timer1 = root.after(1000, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.after_cancel(timer1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_idle(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Set up with callback with no args. + count = 0 + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + idle1 = root.after_idle(callback, 42, 11) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.after_cancel(idle1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_cancel(self): + root = self.root + + def callback(): + nonlocal count + count += 1 + + timer1 = root.after(5000, callback) + idle1 = root.after_idle(callback) + + # No value for id raises a ValueError. + with self.assertRaises(ValueError): + root.after_cancel(None) + + # Cancel timer event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(timer1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', timer1) + + # Cancel same event - nothing happens. + root.after_cancel(timer1) + + # Cancel idle event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(idle1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', idle1) + tests_gui = (MiscTest, ) diff --git a/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst new file mode 100644 index 000000000000..4ebbde4d1946 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst @@ -0,0 +1 @@ +In :mod:`tkinter`, ``after_cancel(None)`` now raises a :exc:`ValueError` instead of canceling the first scheduled function. Patch by Cheryl Sabella.

1 0

bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701)
by Miss Islington (bot) March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/73a43960c7be50e136c5482404980175cb… commit: 73a43960c7be50e136c5482404980175cb99f611 branch: 3.6 author: Miss Islington (bot) <31488909+miss-islington(a)users.noreply.github.com> committer: GitHub <noreply(a)github.com> date: 2018-03-04T03:42:41-08:00 summary: bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701) (cherry picked from commit 74382a3f175ac285cc924a73fd758e8dc3cc41bb) Co-authored-by: Cheryl Sabella <cheryl.sabella(a)gmail.com> files: A Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst M Lib/tkinter/__init__.py M Lib/tkinter/test/test_tkinter/test_misc.py diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index deea791831ed..53bad3fa95ae 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -739,6 +739,7 @@ def after(self, ms, func=None, *args): if not func: # I'd rather use time.sleep(ms*0.001) self.tk.call('after', ms) + return None else: def callit(): try: @@ -762,11 +763,13 @@ def after_cancel(self, id): """Cancel scheduling of function identified with ID. Identifier returned by after or after_idle must be - given as first parameter.""" + given as first parameter. + """ + if not id: + raise ValueError('id must be a valid identifier returned from ' + 'after or after_idle') try: data = self.tk.call('after', 'info', id) - # In Tk 8.3, splitlist returns: (script, type) - # In Tk 8.4, splitlist may return (script, type) or (script,) script = self.tk.splitlist(data)[0] self.deletecommand(script) except TclError: diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 9dc1e37547fc..1d1a3c29f6bc 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -48,6 +48,114 @@ def test_tk_setPalette(self): '^must specify a background color$', root.tk_setPalette, highlightColor='blue') + def test_after(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Without function, sleeps for ms. + self.assertIsNone(root.after(1)) + + # Set up with callback with no args. + count = 0 + timer1 = root.after(0, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.update() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + timer1 = root.after(0, callback, 42, 11) + root.update() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + timer1 = root.after(1000, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.after_cancel(timer1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_idle(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Set up with callback with no args. + count = 0 + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + idle1 = root.after_idle(callback, 42, 11) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.after_cancel(idle1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_cancel(self): + root = self.root + + def callback(): + nonlocal count + count += 1 + + timer1 = root.after(5000, callback) + idle1 = root.after_idle(callback) + + # No value for id raises a ValueError. + with self.assertRaises(ValueError): + root.after_cancel(None) + + # Cancel timer event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(timer1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', timer1) + + # Cancel same event - nothing happens. + root.after_cancel(timer1) + + # Cancel idle event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(idle1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', idle1) + tests_gui = (MiscTest, ) diff --git a/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst new file mode 100644 index 000000000000..4ebbde4d1946 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst @@ -0,0 +1 @@ +In :mod:`tkinter`, ``after_cancel(None)`` now raises a :exc:`ValueError` instead of canceling the first scheduled function. Patch by Cheryl Sabella.

1 0

bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701)
by Serhiy Storchaka March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/74382a3f175ac285cc924a73fd758e8dc3… commit: 74382a3f175ac285cc924a73fd758e8dc3cc41bb branch: master author: Cheryl Sabella <cheryl.sabella(a)gmail.com> committer: Serhiy Storchaka <storchaka(a)gmail.com> date: 2018-03-04T12:41:47+02:00 summary: bpo-32857: Raise error when tkinter after_cancel() is called with None. (GH-5701) files: A Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst M Lib/tkinter/__init__.py M Lib/tkinter/test/test_tkinter/test_misc.py diff --git a/Lib/tkinter/__init__.py b/Lib/tkinter/__init__.py index deea791831ed..53bad3fa95ae 100644 --- a/Lib/tkinter/__init__.py +++ b/Lib/tkinter/__init__.py @@ -739,6 +739,7 @@ def after(self, ms, func=None, *args): if not func: # I'd rather use time.sleep(ms*0.001) self.tk.call('after', ms) + return None else: def callit(): try: @@ -762,11 +763,13 @@ def after_cancel(self, id): """Cancel scheduling of function identified with ID. Identifier returned by after or after_idle must be - given as first parameter.""" + given as first parameter. + """ + if not id: + raise ValueError('id must be a valid identifier returned from ' + 'after or after_idle') try: data = self.tk.call('after', 'info', id) - # In Tk 8.3, splitlist returns: (script, type) - # In Tk 8.4, splitlist may return (script, type) or (script,) script = self.tk.splitlist(data)[0] self.deletecommand(script) except TclError: diff --git a/Lib/tkinter/test/test_tkinter/test_misc.py b/Lib/tkinter/test/test_tkinter/test_misc.py index 9dc1e37547fc..1d1a3c29f6bc 100644 --- a/Lib/tkinter/test/test_tkinter/test_misc.py +++ b/Lib/tkinter/test/test_tkinter/test_misc.py @@ -48,6 +48,114 @@ def test_tk_setPalette(self): '^must specify a background color$', root.tk_setPalette, highlightColor='blue') + def test_after(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Without function, sleeps for ms. + self.assertIsNone(root.after(1)) + + # Set up with callback with no args. + count = 0 + timer1 = root.after(0, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.update() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + timer1 = root.after(0, callback, 42, 11) + root.update() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + timer1 = root.after(1000, callback) + self.assertIn(timer1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.after_cancel(timer1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_idle(self): + root = self.root + + def callback(start=0, step=1): + nonlocal count + count = start + step + + # Set up with callback with no args. + count = 0 + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + # Set up with callback with args. + count = 0 + idle1 = root.after_idle(callback, 42, 11) + root.update_idletasks() # Process all pending events. + self.assertEqual(count, 53) + + # Cancel before called. + idle1 = root.after_idle(callback) + self.assertIn(idle1, root.tk.call('after', 'info')) + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.after_cancel(idle1) # Cancel this event. + self.assertEqual(count, 53) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + + def test_after_cancel(self): + root = self.root + + def callback(): + nonlocal count + count += 1 + + timer1 = root.after(5000, callback) + idle1 = root.after_idle(callback) + + # No value for id raises a ValueError. + with self.assertRaises(ValueError): + root.after_cancel(None) + + # Cancel timer event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', timer1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(timer1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', timer1) + + # Cancel same event - nothing happens. + root.after_cancel(timer1) + + # Cancel idle event. + count = 0 + (script, _) = root.tk.splitlist(root.tk.call('after', 'info', idle1)) + root.tk.call(script) + self.assertEqual(count, 1) + root.after_cancel(idle1) + with self.assertRaises(tkinter.TclError): + root.tk.call(script) + self.assertEqual(count, 1) + with self.assertRaises(tkinter.TclError): + root.tk.call('after', 'info', idle1) + tests_gui = (MiscTest, ) diff --git a/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst new file mode 100644 index 000000000000..4ebbde4d1946 --- /dev/null +++ b/Misc/NEWS.d/next/Library/2018-02-16-14-37-14.bpo-32857.-XljAx.rst @@ -0,0 +1 @@ +In :mod:`tkinter`, ``after_cancel(None)`` now raises a :exc:`ValueError` instead of canceling the first scheduled function. Patch by Cheryl Sabella.

1 0

Daily reference leaks (4243df51fe43): sum=0
by solipsis＠pitrou.net March 4, 2018

March 4, 2018

results for 4243df51fe43 on branch "default" -------------------------------------------- test_collections leaked [-7, 1, 0] memory blocks, sum=-6 test_functools leaked [0, 3, 1] memory blocks, sum=4 test_multiprocessing_fork leaked [2, -1, 1] memory blocks, sum=2 Command line was: ['./python', '-m', 'test.regrtest', '-uall', '-R', '3:3:/home/psf-users/antoine/refleaks/reflogUgVuuC', '--timeout', '7200']

1 0

[3.6] bpo-32981: Fix catastrophic backtracking vulns (GH-5955)
by Benjamin Peterson March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/c9516754067d71fd7429a25ccfcb2141fc… commit: c9516754067d71fd7429a25ccfcb2141fc583523 branch: 3.6 author: Benjamin Peterson <benjamin(a)python.org> committer: GitHub <noreply(a)github.com> date: 2018-03-03T22:59:12-08:00 summary: [3.6] bpo-32981: Fix catastrophic backtracking vulns (GH-5955) * Prevent low-grade poplib REDOS (CVE-2018-1060) The regex to test a mail server's timestamp is susceptible to catastrophic backtracking on long evil responses from the server. Happily, the maximum length of malicious inputs is 2K thanks to a limit introduced in the fix for CVE-2013-1752. A 2KB evil response from the mail server would result in small slowdowns (milliseconds vs. microseconds) accumulated over many apop calls. This is a potential DOS vector via accumulated slowdowns. Replace it with a similar non-vulnerable regex. The new regex is RFC compliant. The old regex was non-compliant in edge cases. * Prevent difflib REDOS (CVE-2018-1061) The default regex for IS_LINE_JUNK is susceptible to catastrophic backtracking. This is a potential DOS vector. Replace it with an equivalent non-vulnerable regex. Also introduce unit and REDOS tests for difflib. Co-authored-by: Tim Peters <tim.peters(a)gmail.com> Co-authored-by: Christian Heimes <christian(a)python.org>. (cherry picked from commit 0e6c8ee2358a2e23117501826c008842acb835ac) files: A Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst M Lib/difflib.py M Lib/poplib.py M Lib/test/test_difflib.py M Lib/test/test_poplib.py M Misc/ACKS diff --git a/Lib/difflib.py b/Lib/difflib.py index 2095a5e517cb..72971d5b87a9 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -1083,7 +1083,7 @@ def _qformat(self, aline, bline, atags, btags): import re -def IS_LINE_JUNK(line, pat=re.compile(r"\s*#?\s*$").match): +def IS_LINE_JUNK(line, pat=re.compile(r"\s*(?:#\s*)?$").match): r""" Return 1 for ignorable line: iff `line` is blank or contains a single '#'. diff --git a/Lib/poplib.py b/Lib/poplib.py index 6bcfa5cfeba3..d8a62c034327 100644 --- a/Lib/poplib.py +++ b/Lib/poplib.py @@ -308,7 +308,7 @@ def rpop(self, user): return self._shortcmd('RPOP %s' % user) - timestamp = re.compile(br'\+OK.*(<[^>]+>)') + timestamp = re.compile(br'\+OK.[^<]*(<.*>)') def apop(self, user, password): """Authorisation diff --git a/Lib/test/test_difflib.py b/Lib/test/test_difflib.py index 156b523c38c1..aaefe6db0291 100644 --- a/Lib/test/test_difflib.py +++ b/Lib/test/test_difflib.py @@ -466,13 +466,33 @@ def _assert_type_error(self, msg, generator, *args): list(generator(*args)) self.assertEqual(msg, str(ctx.exception)) +class TestJunkAPIs(unittest.TestCase): + def test_is_line_junk_true(self): + for line in ['#', ' ', ' #', '# ', ' # ', '']: + self.assertTrue(difflib.IS_LINE_JUNK(line), repr(line)) + + def test_is_line_junk_false(self): + for line in ['##', ' ##', '## ', 'abc ', 'abc #', 'Mr. Moose is up!']: + self.assertFalse(difflib.IS_LINE_JUNK(line), repr(line)) + + def test_is_line_junk_REDOS(self): + evil_input = ('\t' * 1000000) + '##' + self.assertFalse(difflib.IS_LINE_JUNK(evil_input)) + + def test_is_character_junk_true(self): + for char in [' ', '\t']: + self.assertTrue(difflib.IS_CHARACTER_JUNK(char), repr(char)) + + def test_is_character_junk_false(self): + for char in ['a', '#', '\n', '\f', '\r', '\v']: + self.assertFalse(difflib.IS_CHARACTER_JUNK(char), repr(char)) def test_main(): difflib.HtmlDiff._default_prefix = 0 Doctests = doctest.DocTestSuite(difflib) run_unittest( TestWithAscii, TestAutojunk, TestSFpatches, TestSFbugs, - TestOutputFormat, TestBytes, Doctests) + TestOutputFormat, TestBytes, TestJunkAPIs, Doctests) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py index 608eac57f35c..ca9bc6217509 100644 --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -303,9 +303,19 @@ def test_noop(self): def test_rpop(self): self.assertOK(self.client.rpop('foo')) - def test_apop(self): + def test_apop_normal(self): self.assertOK(self.client.apop('foo', 'dummypassword')) + def test_apop_REDOS(self): + # Replace welcome with very long evil welcome. + # NB The upper bound on welcome length is currently 2048. + # At this length, evil input makes each apop call take + # on the order of milliseconds instead of microseconds. + evil_welcome = b'+OK' + (b'<' * 1000000) + with test_support.swap_attr(self.client, 'welcome', evil_welcome): + # The evil welcome is invalid, so apop should throw. + self.assertRaises(poplib.error_proto, self.client.apop, 'a', 'kb') + def test_top(self): expected = (b'+OK 116 bytes', [b'From: postmaster(a)python.org', b'Content-Type: text/plain', diff --git a/Misc/ACKS b/Misc/ACKS index 75c9b3f0563d..9403e110675a 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -351,6 +351,7 @@ Jonathan Dasteel Pierre-Yves David A. Jesse Jiryu Davis Jake Davis +Jamie (James C.) Davis Merlijn van Deen John DeGood Ned Deily diff --git a/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst b/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst new file mode 100644 index 000000000000..9ebabb44f91e --- /dev/null +++ b/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst @@ -0,0 +1,4 @@ +Regexes in difflib and poplib were vulnerable to catastrophic backtracking. +These regexes formed potential DOS vectors (REDOS). They have been +refactored. This resolves CVE-2018-1060 and CVE-2018-1061. +Patch by Jamie Davis.

1 0

[2.7] bpo-32981: Fix catastrophic backtracking vulns (GH-5955)
by Benjamin Peterson March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/e052d40cea15f582b50947f7d906b39744… commit: e052d40cea15f582b50947f7d906b39744dc62a2 branch: 2.7 author: Benjamin Peterson <benjamin(a)python.org> committer: GitHub <noreply(a)github.com> date: 2018-03-03T22:18:17-08:00 summary: [2.7] bpo-32981: Fix catastrophic backtracking vulns (GH-5955) * Prevent low-grade poplib REDOS (CVE-2018-1060) The regex to test a mail server's timestamp is susceptible to catastrophic backtracking on long evil responses from the server. Happily, the maximum length of malicious inputs is 2K thanks to a limit introduced in the fix for CVE-2013-1752. A 2KB evil response from the mail server would result in small slowdowns (milliseconds vs. microseconds) accumulated over many apop calls. This is a potential DOS vector via accumulated slowdowns. Replace it with a similar non-vulnerable regex. The new regex is RFC compliant. The old regex was non-compliant in edge cases. * Prevent difflib REDOS (CVE-2018-1061) The default regex for IS_LINE_JUNK is susceptible to catastrophic backtracking. This is a potential DOS vector. Replace it with an equivalent non-vulnerable regex. Also introduce unit and REDOS tests for difflib. Co-authored-by: Tim Peters <tim.peters(a)gmail.com> Co-authored-by: Christian Heimes <christian(a)python.org>. (cherry picked from commit 0e6c8ee2358a2e23117501826c008842acb835ac) files: A Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst M Lib/difflib.py M Lib/poplib.py M Lib/test/test_difflib.py M Lib/test/test_poplib.py M Misc/ACKS diff --git a/Lib/difflib.py b/Lib/difflib.py index 1c6fbdbedcb7..788a92df3f89 100644 --- a/Lib/difflib.py +++ b/Lib/difflib.py @@ -1103,7 +1103,7 @@ def _qformat(self, aline, bline, atags, btags): import re -def IS_LINE_JUNK(line, pat=re.compile(r"\s*#?\s*$").match): +def IS_LINE_JUNK(line, pat=re.compile(r"\s*(?:#\s*)?$").match): r""" Return 1 for ignorable line: iff `line` is blank or contains a single '#'. diff --git a/Lib/poplib.py b/Lib/poplib.py index b91e5f72d2ca..a238510b38fc 100644 --- a/Lib/poplib.py +++ b/Lib/poplib.py @@ -274,7 +274,7 @@ def rpop(self, user): return self._shortcmd('RPOP %s' % user) - timestamp = re.compile(r'\+OK.*(<[^>]+>)') + timestamp = re.compile(br'\+OK.[^<]*(<.*>)') def apop(self, user, secret): """Authorisation diff --git a/Lib/test/test_difflib.py b/Lib/test/test_difflib.py index 35f2c36ca70a..d8277b79b880 100644 --- a/Lib/test/test_difflib.py +++ b/Lib/test/test_difflib.py @@ -269,13 +269,33 @@ def test_range_format_context(self): self.assertEqual(fmt(3,6), '4,6') self.assertEqual(fmt(0,0), '0') +class TestJunkAPIs(unittest.TestCase): + def test_is_line_junk_true(self): + for line in ['#', ' ', ' #', '# ', ' # ', '']: + self.assertTrue(difflib.IS_LINE_JUNK(line), repr(line)) + + def test_is_line_junk_false(self): + for line in ['##', ' ##', '## ', 'abc ', 'abc #', 'Mr. Moose is up!']: + self.assertFalse(difflib.IS_LINE_JUNK(line), repr(line)) + + def test_is_line_junk_REDOS(self): + evil_input = ('\t' * 1000000) + '##' + self.assertFalse(difflib.IS_LINE_JUNK(evil_input)) + + def test_is_character_junk_true(self): + for char in [' ', '\t']: + self.assertTrue(difflib.IS_CHARACTER_JUNK(char), repr(char)) + + def test_is_character_junk_false(self): + for char in ['a', '#', '\n', '\f', '\r', '\v']: + self.assertFalse(difflib.IS_CHARACTER_JUNK(char), repr(char)) def test_main(): difflib.HtmlDiff._default_prefix = 0 Doctests = doctest.DocTestSuite(difflib) run_unittest( TestWithAscii, TestAutojunk, TestSFpatches, TestSFbugs, - TestOutputFormat, Doctests) + TestOutputFormat, TestJunkAPIs) if __name__ == '__main__': test_main() diff --git a/Lib/test/test_poplib.py b/Lib/test/test_poplib.py index 23d688724b95..d2143759ba66 100644 --- a/Lib/test/test_poplib.py +++ b/Lib/test/test_poplib.py @@ -211,6 +211,16 @@ def test_noop(self): def test_rpop(self): self.assertOK(self.client.rpop('foo')) + def test_apop_REDOS(self): + # Replace welcome with very long evil welcome. + # NB The upper bound on welcome length is currently 2048. + # At this length, evil input makes each apop call take + # on the order of milliseconds instead of microseconds. + evil_welcome = b'+OK' + (b'<' * 1000000) + with test_support.swap_attr(self.client, 'welcome', evil_welcome): + # The evil welcome is invalid, so apop should throw. + self.assertRaises(poplib.error_proto, self.client.apop, 'a', 'kb') + def test_top(self): expected = ('+OK 116 bytes', ['From: postmaster(a)python.org', 'Content-Type: text/plain', diff --git a/Misc/ACKS b/Misc/ACKS index 28b01e8cc7c5..ee2e4fc19393 100644 --- a/Misc/ACKS +++ b/Misc/ACKS @@ -318,6 +318,8 @@ Jonathan Dasteel Pierre-Yves David A. Jesse Jiryu Davis Jake Davis +Jamie (James C.) Davis +Ratnadeep Debnath Merlijn van Deen John DeGood Ned Deily diff --git a/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst b/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst new file mode 100644 index 000000000000..9ebabb44f91e --- /dev/null +++ b/Misc/NEWS.d/next/Security/2018-03-02-10-24-52.bpo-32981.O_qDyj.rst @@ -0,0 +1,4 @@ +Regexes in difflib and poplib were vulnerable to catastrophic backtracking. +These regexes formed potential DOS vectors (REDOS). They have been +refactored. This resolves CVE-2018-1060 and CVE-2018-1061. +Patch by Jamie Davis.

1 0

closes bpo-32980 Remove _PyFrame_Init (GH-5965)
by Benjamin Peterson March 4, 2018

March 4, 2018

https://github.com/python/cpython/commit/7023644e0c310a3006c984318c2c111c46… commit: 7023644e0c310a3006c984318c2c111c468735b4 branch: master author: Thomas Nyberg <tomnyberg(a)gmail.com> committer: Benjamin Peterson <benjamin(a)python.org> date: 2018-03-03T22:06:01-08:00 summary: closes bpo-32980 Remove _PyFrame_Init (GH-5965) files: M Include/pylifecycle.h M Objects/frameobject.c M Python/pylifecycle.c diff --git a/Include/pylifecycle.h b/Include/pylifecycle.h index 659c6df644e3..95dd55b05173 100644 --- a/Include/pylifecycle.h +++ b/Include/pylifecycle.h @@ -148,7 +148,6 @@ PyAPI_FUNC(int) _PySys_EndInit(PyObject *sysdict, _PyMainInterpreterConfig *conf PyAPI_FUNC(_PyInitError) _PyImport_Init(PyInterpreterState *interp); PyAPI_FUNC(void) _PyExc_Init(PyObject * bltinmod); PyAPI_FUNC(_PyInitError) _PyImportHooks_Init(void); -PyAPI_FUNC(int) _PyFrame_Init(void); PyAPI_FUNC(int) _PyFloat_Init(void); PyAPI_FUNC(int) PyByteArray_Init(void); PyAPI_FUNC(_PyInitError) _Py_HashRandomization_Init(const _PyCoreConfig *); diff --git a/Objects/frameobject.c b/Objects/frameobject.c index d308457b7547..b9f380d7b60d 100644 --- a/Objects/frameobject.c +++ b/Objects/frameobject.c @@ -552,14 +552,6 @@ PyTypeObject PyFrame_Type = { _Py_IDENTIFIER(__builtins__); -int _PyFrame_Init() -{ - /* Before, PyId___builtins__ was a string created explicitly in - this function. Now there is nothing to initialize anymore, but - the function is kept for backward compatibility. */ - return 1; -} - PyFrameObject* _Py_HOT_FUNCTION _PyFrame_New_NoTrack(PyThreadState *tstate, PyCodeObject *code, PyObject *globals, PyObject *locals) diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 5db586e15dff..a9b9470c7265 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -690,9 +690,6 @@ _Py_InitializeCore(const _PyCoreConfig *core_config) _Py_ReadyTypes(); - if (!_PyFrame_Init()) - return _Py_INIT_ERR("can't init frames"); - if (!_PyLong_Init()) return _Py_INIT_ERR("can't init longs");

1 0